Instruction: Reshaping order in PyTorch - Fortran-like index ordering

Input:
In numpy, there is an ordering feature for reshaping arrays, by default it is C, but you can specify other ordering like F:
a = np.arange(6).reshape((3, 2))
f = np.reshape(a, (2, 3), order='F') # Fortran-like index ordering
c = np.reshape(a, (2, 3))
print('a= \n', a)
print('f= \n', b)
print('c= \n', c)

the result:
a=
 [[0 1]
 [2 3]
 [4 5]]
f=
 [[0 4 3]
 [2 1 5]]
c=
 [[0 1 2]
 [3 4 5]]

There is no such option in torch.reshape or tensor.view for reshaping in F order.
Is there any way to do that F order reshape in PyTorch? I need everything to be in PyTorch.

Output:
I don't think pytorch has built-in support for this. That said you can achieve the desired result using Tensor.permute. Unfortunately I doubt this will be very efficient since AFAIK permute internally makes a copy of the tensor.
def reshape_fortran(x, shape):
    if len(x.shape) > 0:
        x = x.permute(*reversed(range(len(x.shape))))
    return x.reshape(*reversed(shape)).permute(*reversed(range(len(shape))))

Example usage:
a = torch.arange(6).reshape(3, 2)
f = reshape_fortran(a, (2, 3))
c = a.reshape(2, 3)

which results in
a =
tensor([[0, 1],
        [2, 3],
        [4, 5]])
f =
tensor([[0, 4, 3],
        [2, 1, 5]])
c =
tensor([[0, 1, 2],
        [3, 4, 5]])


Instruction: Creating pytorch Tensors from `torch` or `numpy` vectors

Input:
I am trying to create some testing torch tensors by assembling the dimensions from vectors calculated via basic math functions. As a precursor: assembling Tensors from primitive python arrays does work:
import torch
import numpy as np
torch.Tensor([[1.0, 0.8, 0.6],[0.0, 0.5, 0.75]])
>> tensor([[1.0000, 0.8000, 0.6000],
            [0.0000, 0.5000, 0.7500]])

In addition we can assemble Tensors from numpy arrays
https://pytorch.org/docs/stable/tensors.html:
torch.tensor(np.array([[1, 2, 3], [4, 5, 6]]))
tensor([[ 1,  2,  3],
        [ 4,  5,  6]])

However assembling from calculated vectors is eluding me. Here are some of the attempts made:
X  = torch.arange(0,6.28)
x = X

torch.Tensor([[torch.cos(X),torch.tan(x)]])

torch.Tensor([torch.cos(X),torch.tan(x)])

torch.Tensor([np.cos(X),np.tan(x)])

torch.Tensor([[np.cos(X),np.tan(x)]])

torch.Tensor(np.array([np.cos(X),np.tan(x)]))

All of the above have the following error:

ValueError: only one element tensors can be converted to Python scalars

What is the correct syntax?
Update A comment requested showing x / X.  They're actually set to the same (I changed mind mid-course which to use)
In [56]: x == X
Out[56]: tensor([True, True, True, True, True, True, True])

In [51]:  x
Out[51]: tensor([0., 1., 2., 3., 4., 5., 6.])

In [52]: X
Out[52]: tensor([0., 1., 2., 3., 4., 5., 6.])

Output:
torch.arange returns a torch.Tensor as seen below -
X  = torch.arange(0,6.28)
x
>> tensor([0., 1., 2., 3., 4., 5., 6.])

Similarly, torch.cos(x) and torch.tan(x) returns instances of torch.Tensor
The ideal way to concatenate a sequence of tensors in torch is to use torch.stack
torch.stack([torch.cos(x), torch.tan(x)])

Output
>> tensor([[ 1.0000,  0.5403, -0.4161, -0.9900, -0.6536,  0.2837,  0.9602],
        [ 0.0000,  1.5574, -2.1850, -0.1425,  1.1578, -3.3805, -0.2910]])

If you prefer to concatenate along axis=0, use torch.cat([torch.cos(x), torch.tan(x)]) instead.


Instruction: What does the .numpy() function do?

Input:
I tried searching for the documentation online but I can't find anything that gives me an answer. What does .numpy() function do? The example code given is:
y_true = []
for X_batch, y_batch in mnist_test:
    y_true.append(y_batch.numpy()[0].tolist())

Output: Both in Pytorch and Tensorflow, the .numpy() method is pretty much straightforward. It converts a tensor object into an numpy.ndarray object. This implicitly means that the converted tensor will be now processed on the CPU.


Instruction: Matrix Multiplication with PyTorch

Input:
I'm sorry if this is a basic question, but I am reasonably new to Pytorch and have been trying to perform linear regression. I've been stuck on the following problem for about 3 hours and have tried everything, so I hope you're able to help.
I want to predict grades using a set of four inputs (time studied, travel time, failures and absenses). For my outputs I have grades on 3 tests. There are 395 students in the dataset.
I've listed all my code at the bottom, but I'll paste the relevant part here too:
So far I've created input and output tensors and have created a model for matrix multiplication. Here's my code:
w = torch.randn(395,4, requires_grad=True)
b = torch.randn(4, requires_grad=True)
print(w)
print(b)

def model(x):
    return x @ w.t() + b

predictions = model(inputs)
print(predictions)

I know this isn't the linear regression yet, but I'm really struggling with the matrix multiplication aspect of it. Whenever I run the print(predictions) code I get the following message:
RuntimeError                              Traceback (most recent call last)
<ipython-input-277-c6db141153ac> in <module>
----> 1 preds = model(inputs)
      2 print(preds)

<ipython-input-276-3b94bfbc599e> in model(x)
      1 def model(x):
----> 2     return x @ w.t() + b

RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #3 'mat2' in call to _th_addmm_out

I have a feeling the numbers for w and b are wrong (395,4) and (4), but I have no idea why or what to change them to. Is there any chance someone could point me in the right direction please?
Thanks in advance!
Here's my whole code:
'''
import torch
import numpy as np
from numpy import genfromtxt

data = np.genfromtxt('student-mat.csv', delimiter=',',dtype=float)
data

travel_time = data[1:, 0:1]
study_time = data[1:, 1:2]
failures = data[1:, 2:3]
absenses = data[1:, 3:4]
grade_one = data[1:, 4:5]
grade_two = data[1:, 5:6]
grade_three = data[1:, 6:7]
data_input = data[1:, 0:4]
output = data[1:, 4:7]

inputs = torch.from_numpy(data_input)
outputs = torch.from_numpy(grade_one)
print(inputs)
print(grade_one)

w = torch.randn(395,4, requires_grad=True)
b = torch.randn(4, requires_grad=True)
print(w)
print(b)

def model(x):
    return x @ w.t() + b

preds = model(inputs)
print(preds)

Dataset - https://archive.ics.uci.edu/ml/datasets/Student+Performance

Output: The error message says it all. The tensors involved contain elements of different data types. By default, w and b have elements of type torch.float32, while data_input is a NumPy array with the Python default floating point type, i.e. double. That datatype will be preserved when you convert with from_numpy. Try using dtype=np.float32 in your np.genfromtxt call.


Instruction: How to check if model in on CUDA?

Input:
I would like to check if model is on CUDA. How to do that?
import torch
import torchvision
model = torchvision.models.resnet18()
model.to('cuda')

Seams that model.is_cuda() is not working.

Output:
This code should do it:
import torch
import torchvision
model = torchvision.models.resnet18()
model.to('cuda')
next(model.parameters()).is_cuda

Out:
True

Note there is no is_cuda() method inside nn.Module.
Also note model.to('cuda') is the same as model.cuda() and both are inplace.
On the other hand moving the data.to('cuda') is not inplace and you typically call:
data = data.to('cuda')

to move the data to CUDA.


Instruction: Error installing torchvision (python 3.6)

Input:
I am trying to install torchvision by writing
pip3 install torchvision
but I get the following message
Requirement already satisfied: torchvision in ./venv/lib/python3.6/site-packages (0.7.0)
Requirement already satisfied: numpy in ./venv/lib/python3.6/site-packages (from torchvision) (1.19.2)
Requirement already satisfied: pillow>=4.1.1 in ./venv/lib/python3.6/site-packages (from torchvision) (7.2.0)
Collecting torch==1.6.0 (from torchvision)
  Could not find a version that satisfies the requirement torch==1.6.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.0.post4, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.1.0.post2, 1.2.0, 1.3.0, 1.3.0.post2, 1.3.1, 1.4.0)
No matching distribution found for torch==1.6.0 (from torchvision)

I currently have torch==1.4.0 installed but it is apparently not sufficient to install torchvision. When I tried the code pip3 install torch==1.6.0
I got the following error
Collecting torch==1.6.0
  Could not find a version that satisfies the requirement torch==1.6.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.0.post4, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.1.0.post2, 1.2.0, 1.3.0, 1.3.0.post2, 1.3.1, 1.4.0)
No matching distribution found for torch==1.6.0

Does someone know how I can fix this?

Output: I have solved the issue. So I had torch==1.4.0 installed as this is the latest version acknowledged on my laptop, but for some reason when I tried to install torchvision, it immediately got to the latest version (0.7.0). I found on the pytorch website that torch==1.4.0 is associated to torchvision==0.5.0. So I ran the code pip3 install torch==1.4.0 torchvision==0.5.0 to get torchvision==0.7.0 uninstalled and install 0.5.0. Now it is working fine. Hope this explanation helps


Instruction: Setting the same seed for torch, random number and numpy throughout all the modules

Input:
I am trying to set the same seed throughout all the project.
Below are the parameters I am setting in my main file, in which all other modules will be imported -
seed = 42
os.environ['PYTHONHASHSEED'] = str(seed)
# Torch RNG
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
# Python RNG
np.random.seed(seed)
random.seed(seed)

My project directory looks like below -
├── Combined_Files_without_label5.csv
├── __pycache__
│   ├── dataset.cpython-37.pyc
│   ├── datasets.cpython-37.pyc
│   └── testing.cpython-37.pyc
├── datasets.py
├── import_packages
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-37.pyc
│   │   ├── dataset.cpython-37.pyc
│   │   ├── dataset_class.cpython-37.pyc
│   │   ├── dataset_partition.cpython-37.pyc
│   │   └── visualising.cpython-37.pyc
│   ├── dataset_class.py
│   ├── dataset_partition.py
│   └── visualising.py
├── main.py

Now, the problem is I am importing the module from dataset_partition.py and the function  needs a seed value there. e.g -
    df_train, df_temp, y_train, y_temp = train_test_split(X,
                                                      y,
                                                      stratify=y,
                                                      test_size=(1.0 - frac_train), # noqa
                                                      random_state=seed)

Now, my questions are, 1)If I just remove the random_state parameter from the above statement so will it take the seed from my main file? If not, then how to set it?
2)Does all the other function which requires seed like torch.manual.seed, torch.cuda.manual_seed(seed) will behave in the same way?(If not, then how to resolve it)

Output:
1)If I just remove the random_state parameter from the above statement
so will it take the seed from my main file?

Yes, as the docs for default (None) value say:

Use the global random state instance from numpy.random. Calling the
function multiple times will reuse the same instance, and will produce
different results.

As you are using this in __init__ I suppose, it will be run before any other function you use from your package and you are fine.

2)Does all the other function which requires seed
like torch.manual.seed, torch.cuda.manual_seed(seed) will behave in
the same way?

Yes, those will set global seed for Python and PyTorch to use and you are also fine here.


Instruction: How to get only specific classes from PyTorch's FashionMNIST dataset?

Input:
The FashionMNIST dataset has 10 different output classes. How can I get a subset of this dataset with only specific classes? In my case, I only want images of sneaker, pullover, sandal and shirt classes (their classes are 7,2,5 and 6 respectively).
This is how I load my dataset.
train_dataset_full = torchvision.datasets.FashionMNIST(data_folder, train = True, download = True, transform = transforms.ToTensor())
The approach I’ve followed is below.
Iterate through the dataset, one by one, then compare the 1st element (i.e. class) in the returned tuple to my required class. I’m stuck here. If the value returned is true, how can I append/add this observation to an empty dataset?
sneaker = 0
pullover = 0
sandal = 0
shirt = 0
for i in range(60000):
    if train_dataset_full[i][1] == 7:
        sneaker += 1
    elif train_dataset_full[i][1] == 2:
        pullover += 1
    elif train_dataset_full[i][1] == 5:
        sandal += 1
    elif train_dataset_full[i][1] == 6:
        shirt += 1

Now, in place of sneaker += 1, pullover += 1, sandal += 1 and shirt += 1 I want to do something like this empty_dataset.append(train_dataset_full[i]) or something similar.
If the above approach is incorrect, please suggest another method.

Output:
Finally found the answer.
dataset_full = torchvision.datasets.FashionMNIST(data_folder, train = True, download = True, transform = transforms.ToTensor())
# Selecting classes 7, 2, 5 and 6
idx = (dataset_full.targets==7) | (dataset_full.targets==2) | (dataset_full.targets==5) | (dataset_full.targets==6)
dataset_full.targets = dataset_full.targets[idx]
dataset_full.data = dataset_full.data[idx]


Instruction: pytorch LSTM model not learning

Input:
I created a simple LSTM model to predict Uniqlo closing price. The problem is, my model doesn't seem to learn anything. This is the link to my notebook
This is my model creation class (I tried relu activation function before, get the same outcome):
class lstm(torch.nn.Module):
  def __init__(self,hidden_layers):
    super(lstm,self).__init__()
    self.hidden_layers = hidden_layers
    self.lstm = torch.nn.LSTM(input_size = 2,hidden_size = 100,num_layers = self.hidden_layers,batch_first=True)
    self.hidden1 = torch.nn.Linear(100,80)
    self.dropout1 = torch.nn.Dropout(0.1)
    self.hidden2 = torch.nn.Linear(80,60)
    self.dropout2 = torch.nn.Dropout(0.1)
    self.output = torch.nn.Linear(60,1)

  def forward(self,x):
    batch = len(x)
    h = torch.randn(self.hidden_layers,batch,100).requires_grad_().cuda()
    c = torch.randn(self.hidden_layers,batch,100).requires_grad_().cuda()

    x,(ho,co)= self.lstm(x.view(batch,10,2),(h.detach(),c.detach()))
    x = torch.reshape(x[:,-1,:],(batch,-1))
    x = self.hidden1(x)
    x = torch.nn.functional.tanh(x)
    x = self.dropout1(x)
    x = self.hidden2(x)
    x = torch.nn.functional.tanh(x)
    x = self.dropout2(x)
    x = self.output(x)
    return x

model = lstm(10)

This is my training loss plot:
training loss
This is my validation loss plot:
validation loss
This is my ground truth (blue) vs prediction (orange):
ground truth vs prediction
Can anyone please point out what did I do wrongly?

Output:
You train scaler with whole data. This is not a good strategy. You should use only the training data.
You don't have to scale targets. Use it directly or apply log function or use returns.
About hidden state and cell memory, why do you kept track gradients and detach them just after? You don't have to detach hidden state and cell memory when feeding lstm layer because it participates of back propagation.
If I understand what you did, you predict the next close price using the last 10 open prices and volumes. I don't think you can get good results with this configuration. You should formalize the problem better.


Instruction: Understanding behavior of index_put in PyTorch

Input:
I am trying to understand the behavior of index_put in PyTorch, but the document is not clear to me.
Given
a = torch.zeros(2, 3)
a.index_put([torch.tensor(1, 0), torch.tensor([1, 1])], torch.tensor(1.))

it returns
tensor([[1., 1., 0.],
       [0., 0., 0.])

While given
a = torch.zeros(2, 3)
a.index_put([torch.tensor(0, 0), torch.tensor([1, 1])], torch.tensor(1.))

it returns
tensor([[0., 1., 0.],
       [0., 0., 0.])

I am wondering what the rule of index_put on earth? What if I want to put three values to a, such that it returns
tensor([0., 1., 1.,],
       [0., 1., 0.])

Any help is appreciated!

Output:
I copied your examples here with argument names inserted, fixed brackets and correct output(yours was swapped):
a.index_put(indices=[torch.tensor([1, 0]), torch.tensor([1, 1])], values=torch.tensor(1.))

tensor([[0., 1., 0.],
        [0., 1., 0.]])

a.index_put(indices=[torch.tensor([0, 0]), torch.tensor([0, 1])], values = torch.tensor(1.))

tensor([[1., 1., 0.],
        [0., 0., 0.]]

What this method does is inserting value(s) into locations in the original a tensor indicated by indices. indices is a list of x coordinates of insertions and y coordinates of insertions. values may be single value or a 1d tensor.
to obtain the desired output use:
a.index_put(indices=[torch.tensor([0,0,1]), torch.tensor([1, 2, 1])], values=torch.tensor(1.))

tensor([[0., 1., 1.],
        [0., 1., 0.]])

moreover, you can pass multiple values in values argument to insert them into the indicated positions:
a.index_put(indices=[torch.tensor([0,0,1]), torch.tensor([1, 2, 1])], values=torch.tensor([1., 2., 3.]))

tensor([[0., 1., 2.],
        [0., 3., 0.]])


Instruction: Output activation function for tensors for multilabel classification

Input:
My expected label is a tensor like tensor([[0, 1, 0, 1], [1, 1, 1, 0]]).
The output from my model (note that I stopped training just to obtain the values so they do not accurately represent the activations of neurons) is also a tensor like tensor([[-10.6964, -13.8998, 0.8348, -45.7040], [-10.3260, -13.8385, -9.2342, -5.3424]])
Am I wrong in using BCEWithLogitsLoss directly on the outputs and labels? Would I need to convert the output tensor to a binary one similar to the expected label prior to using BCEWithLogitsLoss? I understand that BCEWithLogitsLoss is just BCELoss + Sigmoid activation. How can I obtain values of the type of the expected label tensor and what loss should I use in that case?

Output:
In case of multi-label classification BCELoss is a common choice. BCEWithLogitsLoss works directly on the outputs and labels in the sense that it expects the logits as one input and the class (0 or 1) as the second input.
You can find more details regarding this loss and the expected label tensor here:
BCEWithLogitsLoss


Instruction: Understanding ‘backward()’: How to code the Pytorch function ‘.backward()’ from scratch?

Input: I’m a newbie learning Deep Learning, I’m stuck trying to understand what ‘.backward()’ from Pytorch does since it does pretty much most the work there. Therefor, I’m trying to understand what backward function does in detail so, I’m going to try to code what the function does step by step. Any resource that you can recommend me (Book, video, GitHub repo) to start coding the function? Thank you for time and hopefully for your help.

Output:
backward() is calculating the gradients with respect to (w.r.t.) graph leaves.
grad() function is more general it can calculate the gradients w.r.t. any inputs (leaves included).
I implemented the grad() function, some time ago, you may check this. It uses the power of Automatic Differentiation (AD).
import math
class ADNumber:

    def __init__(self,val, name=""):
        self.name=name
        self._val=val
        self._children=[]

    def __truediv__(self,other):
        new = ADNumber(self._val / other._val, name=f"{self.name}/{other.name}")
        self._children.append((1.0/other._val,new))
        other._children.append((-self._val/other._val**2,new)) # first derivation of 1/x is -1/x^2
        return new

    def __mul__(self,other):
        new = ADNumber(self._val*other._val, name=f"{self.name}*{other.name}")
        self._children.append((other._val,new))
        other._children.append((self._val,new))
        return new

    def __add__(self,other):
        if isinstance(other, (int, float)):
            other = ADNumber(other, str(other))
        new = ADNumber(self._val+other._val, name=f"{self.name}+{other.name}")
        self._children.append((1.0,new))
        other._children.append((1.0,new))
        return new

    def __sub__(self,other):
        new = ADNumber(self._val-other._val, name=f"{self.name}-{other.name}")
        self._children.append((1.0,new))
        other._children.append((-1.0,new))
        return new

    @staticmethod
    def exp(self):
        new = ADNumber(math.exp(self._val), name=f"exp({self.name})")
        self._children.append((self._val,new))
        return new

    @staticmethod
    def sin(self):
        new = ADNumber(math.sin(self._val), name=f"sin({self.name})")
        self._children.append((math.cos(self._val),new)) # first derivative is cos
        return new

    def grad(self,other):
        if self==other:
            return 1.0
        else:
            result=0.0
            for child in other._children:
                result+=child[0]*self.grad(child[1])
            return result

A = ADNumber # shortcuts
sin = A.sin
exp = A.exp

def print_childs(f, wrt): # with respect to
    for e in f._children:
        print("child:", wrt, "->" , e[1].name, "grad: ", e[0])
        print_child(e[1], e[1].name)

x1 = A(1.5, name="x1")
x2 = A(0.5, name="x2")
f=(sin(x2)+1)/(x2+exp(x1))+x1*x2

print_childs(x2,"x2")
print("\ncalculated gradient for the function f with respect to x2:", f.grad(x2))

Out:
child: x2 -> sin(x2) grad:  0.8775825618903728
child: sin(x2) -> sin(x2)+1 grad:  1.0
child: sin(x2)+1 -> sin(x2)+1/x2+exp(x1) grad:  0.20073512936690338
child: sin(x2)+1/x2+exp(x1) -> sin(x2)+1/x2+exp(x1)+x1*x2 grad:  1.0
child: x2 -> x2+exp(x1) grad:  1.0
child: x2+exp(x1) -> sin(x2)+1/x2+exp(x1) grad:  -0.05961284871202578
child: sin(x2)+1/x2+exp(x1) -> sin(x2)+1/x2+exp(x1)+x1*x2 grad:  1.0
child: x2 -> x1*x2 grad:  1.5
child: x1*x2 -> sin(x2)+1/x2+exp(x1)+x1*x2 grad:  1.0

calculated gradient for the function f with respect to x2: 1.6165488003791766


Instruction: How can I convert SmoothedValue into float for plotting using pyplot?

Input:
I am trying to plot the loss vs epoch graph from my neural network using matplotlib.pyplot, but I am encountering a problem. After each epoch, the losses are gathered in an array and are SmoothedValue objects (from utils.py). Right now I am trying to get the loss after each epoch and storing it in an array so that later I can use it as an axis for pyplot. I do this the following way:
for epoch in range(num_epochs):
    epoch_list.append(epoch)
    # train for one epoch, printing every 10 iterations
    metric_logger = train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)

    loss_list = metric_logger.meters.get('loss')
    loss_axis.append(loss_list)

    print("Loss: ", loss_list.value)
    print("Loss: ", loss_list.deque)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    coco_evaluator = evaluate(model, data_loader_test, device=device)

After training in each epoch, I get the value corresponding to the key 'loss' in the dictionary metric_logger.meters, which I then append to my array loss_axis. However, the elements in metric_logger.meters are of the type SmoothedValue, which then my program can't interpret to plot it. How can I convert this Smoothed Value type to float so that I can plot it?

Output: It should work if you replace loss_axis.append(loss_list) with loss_axis.append(loss_list.value).


Instruction: pytorch - how to troubleshoot device (cpu \ gpu) settings of tensors \ models

Input:
I have a torch model that i'm trying to port from CPU do be device independent.
setting the device parameter when creating tensors, or calling model.to(device) to move a full model to the target device solves part of the problem, but there are some "left behind" tensors (like variables created during the forward call)
is there a way to detect these without using an interactive debugger?
something like tracing tensor creation to allow discover of tensors that are created on the wrong device?

Output:
You could check the garbage collector:
import gc
import torch

s = torch.tensor([2], device='cuda:0')
t = torch.tensor([1])
for obj in gc.get_objects():
    if torch.is_tensor(obj):
        print(obj)

Output:
tensor([2], device='cuda:0')
tensor([1])


Instruction: Is it possible to train a neural netowrk using a loss function on unseen data (data different from input data)?

Input:
Normally, a loss function may be defined as
L(y_hat, y) or L(f(X), y), where f is the neural network, X is the input data, y is the target.
Is it possible to implement (preferably in PyTorch) a loss function that depends not only on the input data X, but also on X' (X != X)?
For example, let's say I have a neural network f, input data (X,y) and X'. Can I construct a loss function such that

f(X) is as close as possible to y, and also
f(X') > f(X)?

The first part can be easily implemented (PyTorch: nn.MSELoss()), the second part seems to be way harder.
P.S: this question is a reformulation of Multiple regression while avoiding line intersections using neural nets, which was closed. In the original data, input data and photos with a theoretical example are available.

Output:
Yes it is possible.
For instance, you can add a loss term using ReLU as follows:
loss = nn.MSELoss()(f(X),y) + lambd * nn.ReLU()(f(X)-f(X'))

where lambd is a hyperparameter.
Note that this corresponds to f(X') >= f(X) but it's easily modifiable to f(X') > f(X) by adding an epsilon<0 (small enough in absolute value) constant inside the ReLU.


Instruction: BCEWithLogitsLoss: Trying to get binary output for predicted label as a tensor, confused with output layer

Input: Each element of my dataset has a multi-label tensor like [1, 0, 0, 1] with varying combinations of 1's and 0's. In this scenario, since I have 4 tensors, I have the output layer of my neural network to be 4. In doing so with BCEWithLogitsLoss, I obtain an output tensor like [3, 2, 0, 0] when I call model(inputs) which is in the range of (0, 3) as I specified with the output layer to have 4 output neurons. This does not match the format of what the target is expected to be, although when I change the number of output neurons to 2, I get a shape mismatch error. What needs to be done to fix this?

Output:
When using BCEWithLogitsLoss you make a 1D prediction per output binary label.
In your example, you have 4 binary labels to predict, and therefore, your model outputs 4d vector, each entry represents the prediction of one of the binary labels.
Using BCEWithLogitsLoss you implicitly apply Sigmoid to your outputs:

This loss combines a Sigmoid layer and the BCELoss in one single class.

Therefore, if you want to get the predicted probabilities of your model, you need to add a torch.sigmoid on top of your prediction. The sigmoid function will convert your predicted logits to probabilities.


Instruction: torch.einsum doesn't accept float tensors?

Input:
I get the following error:

RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #3 'mat2' in call to _th_addmm_out

I use torch.einsum as follows:
mu = torch.einsum('ijl, akij -> akl', idxs, activation_map)
I don't understand this, as in the documentation they are using float tensors too (https://pytorch.org/docs/stable/generated/torch.einsum.html). Also choosing a long tensor is no option, as all values in activation_map are between 0 and 1.

Output:
It seems like your first argument, idxs is of type Long.
All input tensors to torch.einsum should be Float.


Instruction: How can I import my CNN-model in another code?

Input:
This is my "model", I just added a piece of it here for understanding purpose. I want this model to be imported in another piece of code named as "trainer".
import torch.nn as nn
import torch

class Signet(nn.Module):
    def __init__(self, num_users=10):
        super(Signet, self).__init__()
        self.num_classes = num_users
        self.forward_util_layer = {}

This is another piece of code. Here "Model" is what I want to import like I mentioned above.
import model

class Trainer:
    def __init__(self, trainData, testData, num_users=10):
        self.trainData = trainData
        self.testData = testData

What are the ways to import it. Where to save my model.ipynb, so that my trainer.ipynb can import it. Or if there is any way to call it directly in my trainer.ipynb code?

Output:
Move your model code to model.py file. Keep that file in the same directory as your trainer.ipynb
Now, to load the model in the ipynb file, follow these steps.
import sys
sys.path.insert(0, '/path/to/model/folder')
from model import Signet
m = Signet(15)


Instruction: Best way to define a scalar using nn.Parameter in Pytorch

Input:
In my CNN at some stage I want to multiply a feature map with some scalar which should be learnt by the network. Which of the following is the best way to do it or all are same? The scalar has to be initialised to 5.
# Method 1
def __init__(self):
    super(..., self).__init__()
    ...
    ...
    alpha = nn.Parameter(5)
    ...

def forward(self, x):
    ...
    x = x * alpha
    return x

# Method 2
def __init__(self):
    super(..., self).__init__()
    ...
    ...
    alpha = nn.Parameter(torch.tensor(5))
    ...

def forward(self, x):
    ...
    x = x * alpha
    return x

# Method 3
def __init__(self):
    super(..., self).__init__()
    ...
    ...
    alpha = nn.Parameter(torch.ones(1)*5)
    ...

def forward(self, x):
    ...
    x = x * alpha
    return x

If all are same I would prefer Method 1 and let CNN learn the appropriate multiplier alpha for the feature map x. I hope in all cases alpha will be a float32 tensor initialised to 5. I am using PyTorch 1.3.1
Yours sincerely,
Mohit

Output:
The third option will work since the parameter constructor needs a float. Parameters are updated with the optimizer, so they need to have gradients, apart from buffers.
Buffers are managed by yourself, not with the optimizer.
You may play with this experimental code.
BS=2
class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.weights = nn.Parameter(torch.randn(BS, 2))
        self.bias = nn.Parameter(torch.zeros(BS))
        self.alpha = nn.Parameter(torch.tensor(5.))

    def forward(self, x):
        return x @ self.weights + self.bias

m=M()
m.parameters()
list(m.parameters())

Out:
[Parameter containing:
 tensor([[-0.5627,  2.3067],
         [ 1.3821, -0.1893]], requires_grad=True), Parameter containing:
 tensor([0., 0.], requires_grad=True), Parameter containing:
 tensor(5., requires_grad=True)]

Here I directly set the value 5. to the parameter alpha, and I added a few more parameters for fun.
You could also do as explained by Shai:
self.register_parameter(name='alpha', param=torch.nn.Parameter(torch.tensor(5.)))

You may asked why we have nn.Module.register_parameter, why don't we just use the nn.Parameter() approach?
nn.Module.register_parameter takes the name and tensor and first checks if the name is in the dictionary of the module. While nn.Parameter() doesn't have such a check.


Instruction: pytorch conv2d with weights

Input:
I have these 2 variables:
result- tensor 1 X 251 X 20
kernel - tensor 1 X 10 X 10

when I run the command:
from torch.nn import functional as F
result = F.conv2d(result, kernel)

I get the error:
RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]

I am not giving any stride, what am I doing wrong?

Output:
import torch
import torch.nn.functional as F
image = torch.rand(16, 3, 32, 32)
filter = torch.rand(1, 3, 5, 5)
out_feat_F = F.conv2d(image, filter,stride=1, padding=0)
print(out_feat_F.shape)

Out:
torch.Size([16, 1, 28, 28])

Which is equivalent with:
import torch
import torch.nn
image = torch.rand(16, 3, 32, 32)
conv_filter = torch.nn.Conv2d(in_channels=3, out_channels=1, kernel_size=5, stride=1, padding=0)
output_feature = conv_filter(image)
print(output_feature.shape)

Out:
torch.Size([16, 1, 28, 28])

Padding is by default 0, stride is by default 1.
The filter last two dimensions in the first example correspond to the
kernel size in the second example.
kernel_size=5 is the same as kernel_size=(5,5).


Instruction: How to calculate the 3x3 covariance matrix for RGB values across an image dataset?

Input:
I need to calculate the covariance matrix for RGB values across an image dataset, and then apply Cholesky decomposition to the final result.
The covariance matrix for RGB values is a 3x3 matrix M, where M_(i, i) is the variance of channel i and M_(i, j) is the covariance between channels i and j.
The end result should be something like this:
([[0.26, 0.09, 0.02],
[0.27, 0.00, -0.05],
[0.27, -0.09, 0.03]])

I'd prefer to stick to PyTorch functions even though Numpy has a Cov function.
I attempted to recreate the numpy Cov function in PyTorch here based on other cov implementations and clones:
def pytorch_cov(tensor, tensor2=None, rowvar=True):
    if tensor2 is not None:
        tensor = torch.cat((tensor, tensor2), dim=0)
    tensor = tensor.view(1, -1) if tensor.dim() < 2 else tensor
    tensor = tensor.t() if not rowvar and tensor.size(0) != 1 else tensor
    tensor = tensor - torch.mean(tensor, dim=1, keepdim=True)
    return 1 / (tensor.size(1) - 1) * tensor.mm(tensor.t())

def cov_vec(x):
    c = x.size(0)
    m1 = x - torch.sum(x, dim=[1],keepdims=True)/ c
    out = torch.einsum('ijk,ilk->ijl',m1,m1)  / (c - 1)
    return out

The dataset loading would be like this:
dataset = torchvision.datasets.ImageFolder(data_path)
loader = torch.utils.data.DataLoader(dataset)

for images, _ in loader:
    batch_size = images.size(0)
    ...

For the moment I'm just experimenting with images created with torch.randn(batch_size, 3, height, width).
Edit:
I'm attempting to replicate the matrix from Tensorflow's Lucid here, and somewhat explained on distill.pub here.
Second Edit:
In order to make the output resemble the example one, you have to do this instead of using Cholesky:
rgb_cov_tensor = rgb_cov_tensor / len(loader.dataset)
U,S,V = torch.svd(rgb_cov_tensor)
epsilon = 1e-10
svd_sqrt = U @ torch.diag(torch.sqrt(S + epsilon))

The resulting matrix can then be used to perform color decorrelation, which is useful for visualizing features (DeepDream). I've implemented it in my project here.

Output:
Here is a function for computing the (unbiased) sample covariance matrix on a 3 channel image, named rgb_cov. Cholesky decomposition is straightforward with torch.cholesky:
import torch
def rgb_cov(im):
    '''
    Assuming im a torch.Tensor of shape (H,W,3):
    '''
    im_re = im.reshape(-1, 3)
    im_re -= im_re.mean(0, keepdim=True)
    return 1/(im_re.shape[0]-1) * im_re.T @ im_re

#Test:
im = torch.randn(50,50,3)
cov = rgb_cov(im)
L_cholesky = torch.cholesky(cov)


Instruction: Any reason to save a pretrained BERT tokenizer?

Input:
Say I am using tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True), and all I am doing with that tokenizer during fine-tuning of a new model is the standard tokenizer.encode()
I have seen in most places that people save that tokenizer at the same time that they save their model, but I am unclear on why it's necessary to save since it seems like an out-of-the-box tokenizer that does not get modified in any way during training.

Output:
In your case, if you are using tokenizer only to tokenize the text (encode()), then you need not have to save the tokenizer. You can always load the tokenizer of the pretrained model.
However, sometimes you may want to use the tokenizer of the pretrained model, then add new tokens to it's vocabulary, or redefine the special symbols such as '[CLS]', '[MASK]', '[SEP]', '[PAD]' or any such special tokens. In this case, since you have made the changes to the tokenizer, it will be useful to save the tokenizer for the future use.


Instruction: How to compute hessian matrix for all parameters in a network in pytorch?

Input:
Suppose vector \theta is all the parameters in a neural network, I wonder how to compute hessian matrix for \theta in pytorch.
Suppose the network is as follows:
class Net(Module):
    def __init__(self, h, w):
        super(Net, self).__init__()
        self.c1 = torch.nn.Conv2d(1, 32, 3, 1, 1)
        self.f2 = torch.nn.Linear(32 * h * w, 5)

    def forward(self, x):
        x = self.c1(x)
        x = x.view(x.size(0), -1)
        x = self.f2(x)
        return x

I know the second derivative can be calculated by calling torch.autograd.grad() twice, but the parameters in pytorch is organized by net.parameters(), and I don't know how to compute the hessian for all parameters.
I have tried to use torch.autograd.functional.hessian() in pytorch 1.5 as follows:
import torch
import numpy as np
from torch.nn import Module
import torch.nn.functional as F

class Net(Module):
    def __init__(self, h, w):
        super(Net, self).__init__()
        self.c1 = torch.nn.Conv2d(1, 32, 3, 1, 1)
        self.f2 = torch.nn.Linear(32 * h * w, 5)

    def forward(self, x):
        x = self.c1(x)
        x = x.view(x.size(0), -1)
        x = self.f2(x)
        return x

def func_(a, b c, d):
    p = [a, b, c, d]
    x = torch.randn(size=[8, 1, 12, 12], dtype=torch.float32)
    y = torch.randint(0, 5, [8])
    x = F.conv2d(x, p[0], p[1], 1, 1)
    x = x.view(x.size(0), -1)
    x = F.linear(x, p[2], p[3])
    loss = F.cross_entropy(x, y)
    return loss

if __name__ == '__main__':
    net = Net(12, 12)

    h = torch.autograd.functional.hessian(func_, tuple([_ for _ in net.parameters()]))
    print(type(h), len(h))

h is a tuple, and the results are in strange shape. For example, the shape of \frac{\delta Loss^2}{\delta c1.weight^2} is [32,1,3,3,32,1,3,3]. It seems like I can combine them into a complete H, but I don't know which part it is in the whole Hessian Matrix and the corresponding order.

Output:
Here is one solution, I think it's a little too complex but could be instructive.
Considering about these points:

First, about torch.autograd.functional.hessian() the first argument must be a function, and the second argument should be a tuple or list of tensors. That means we cannot directly pass a scalar loss to it. (I don't know why, because I think there is no large difference between a scalar loss or a function that returns a scalar)
Second, I want to obtain a complete Hessian matrix, which is the second derivative of all parameters, and it should be in an appropriate order.

So here is the solution:
import torch
import numpy as np
from torch.nn import Module
import torch.nn.functional as F

class Net(Module):
    def __init__(self, h, w):
        super(Net, self).__init__()
        self.c1 = torch.nn.Conv2d(1, 32, 3, 1, 1)
        self.f2 = torch.nn.Linear(32 * h * w, 5)

    def forward(self, x):
        x = self.c1(x)
        x = x.view(x.size(0), -1)
        x = self.f2(x)
        return x

def haha(a, b, c, d):
    p = [a.view(32, 1, 3, 3), b, c.view(5, 32 * 12 * 12), d]
    x = torch.randn(size=[8, 1, 12, 12], dtype=torch.float32)
    y = torch.randint(0, 5, [8])
    x = F.conv2d(x, p[0], p[1], 1, 1)
    x = x.view(x.size(0), -1)
    x = F.linear(x, p[2], p[3])
    loss = F.cross_entropy(x, y)
    return loss

if __name__ == '__main__':
    net = Net(12, 12)

    h = torch.autograd.functional.hessian(haha, tuple([_.view(-1) for _ in net.parameters()]))

    # Then we just need to fix tensors in h into a big matrix

I build a new function haha that works in the same way with the neural network Net. Notice that arguments a, b, c, d are all expanded into one-dimensional vectors, so that the shapes of tensors in h are all two dimensional, in good order and easy to be combined into a large hessian matrix.
In my example, the shapes of tensors in h is
# with relation to c1.weight and c1.weight, c1.bias, f2.weight, f2.bias
[288,288]
[288,32]
[288,23040]
[288,5]

# with relation to c2.bias and c1.weight, c1.bias, f2.weight, f2.bias
[32, 288]
[32, 32]
[32, 23040]
[32, 5]
...

So it is easy to see the meaning of the tensors and which part it is. All we need to do is to allocate a (288+32+23040+5)*(288+32+23040+5) matrix and fix the tensors in h into the corresponding locations.
I think the solution still could be improved, like we don't need to build a function works the same way with neural network, and transform the shape of parameters twice. But for now I don't have better ideas, if there is any better solution, please let me know.


Instruction: Creating a generator over a zarr array with start and end for pytorch dataloader

Input:
I'm working on a pytorch project where my data is saved in zarr.
Random access on zarr is costly, but thanks to zarr using a blockwise cache, iteration is really quick. To harness this fact, I use an IterableDataset together with multiple workers:
class Data(IterableDataset):
    def __init__(self, path, start=None, end=None):
        super(Data, self).__init__()
        store = zarr.DirectoryStore(path)
        self.array = zarr.open(store, mode='r')

        if start is None:
            start = 0
        if end is None:
            end = self.array.shape[0]

        assert end > start

        self.start = start
        self.end = end

    def __iter__(self):
        return islice(self.array, self.start, self.end)

The issue is that self.array has on the order of 10e9 rows and for consecutive workers, as self.start and self.end naturally get bigger, creating the generators like itertools.islice(array, start, end) takes a significant time out of my training/validation processes, because islice still has to iterate over the unneeded elements until it gets to start. Once a generator is created per each worker, this works like a charm, but to get there takes too long.
Is there a better way to create such a generator? Or maybe there's a smarter way to use zarr in pytorch?

Output:
Update: As of March 2021 this change has been merged into zarr.
I took a small dive into zarr and it looks like this will most easily be enabled from inside zarr. I have opened an issue here, in the meantime I made a fork of zarr that implements the function array.islice(start, end).
The dataset __iter__ method then looks like this:
def __iter__(self):
    return self.array.islice(self.start, self.end)


Instruction: How to prepare this PyTorch official ImageNet example?

Input:
This is a technical question on preparing a dataset.
I'm trying to follow this official example
https://github.com/pytorch/examples/tree/master/imagenet
but I cannot even start with because I don't understand the requirements. It says

Install PyTorch (pytorch.org)
pip install -r requirements.txt
Download the ImageNet dataset from http://www.image-net.org/
Then, and move validation images to labeled subfolders, using the following shell script

For the first requirement, I'm working on Colab, so I don't think I need to install PyTorch again on my local pc.
The second one doesn't work, as there's obviously no module named "requirements.txt". This is where I'm beginning to realize there's something on this git repo that I completely don't understand how to use. Anyway, I could just open the text file from the git repo directly, and it just says use torch and torchvision. Okay, I have no problem importing them.
The third requirement. So I went to ImageNet website and signed the agreement for the research use. Now the requirement tells me to download THE ImageNet data, but I see bunch of various options there (like by published years, purposes like for a competition, resolution, etc.). Which one is THE DATASET?
I'm new to PyTorch, and I think I'm missing some protocol about how the PyTorch dev community provides examples via this way...
Any help will be appreciated. Thank you.

Output:
there's obviously no module named "requirements.txt"

It's the requirements.txt file in that repo. You can add package names in a file such as this and install all packages at once using pip, that's why pip install -r requirements.txt. Of course, since it only contains torch and torvision, you don't need to install it as these are already installed on google colab.

Which one is THE DATASET?

I can't access this page without signing up, though you can download any dataset (of any year etc), the important thing is that in order to train it using pytorch using Imagefolder api (which is the one used in the repo you mentioned), its structure should be like this:
train/
    dog/
        xxx.png
        xxy.png
    cat/
        xxz.png
val/
    ...

You can use the script they mentioned for Imagenet data to do so.
If you're just getting started with pytorch, I'd advise you to go through pytorch tutorials such as this one.


Instruction: Pytorch BERT: Misshaped inputs

Input:
I am running into issues of evaluating huggingface's BERT model ('bert-base-uncased') on large input sequences.
model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
token_ids = [101, 1014, 1016, ...] # len(token_ids) == 33286
token_tensors = torch.tensor([token_ids]) # shape == [1, 33286]
segment_tensors = torch.tensor([[1] * len(token_ids)]) # shape == [1, 33286]
model(token_tensors, segment_tensors)

Traceback
self.model(token_tensors, segment_tensors)
  File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/.../python3.8/site-packages/transformers/modeling_bert.py", line 824, in forward
    embedding_output = self.embeddings(
  File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/.../python3.8/site-packages/transformers/modeling_bert.py", line 211, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (33286) must match the size of tensor b (512) at non-singleton dimension 1

I noticed that model.embeddings.positional_embeddings.weight.shape == (512, 768). I.e. when I restrict the input size to model(token_tensors[:, :10], segment_tensors[:, :10]) it works. I am misunderstanding how the the token_tensors and segment_tensors should be shaped. I thought they should be sized (batch_size, sequence_length)
Thanks for the help

Output: I just discovered that pretrained BERT models from huggingface have a maximum input length of 512 ( https://github.com/huggingface/transformers/issues/225 )


Instruction: Pytorch weighted Tensor

Input:
I'm porting a little bit complex TF2 code to Pytorch. Since TF2 does not distinguish Tensor and numpy array, it was straightforward on it. However, I feel like I came back to the TF1 era when I encountered several errors saying 'you cannot mix Tensor and numpy array here in Pytorch!'. Here is the original TF2 code:
def get_weighted_imgs(points, centers, imgs):
  weights = np.array([[tf.norm(p - c) for c in centers] for p in points], dtype=np.float32)
  weighted_imgs = np.array([[w * img for w, img in zip(weight, imgs)] for weight in weights])

  weights = tf.expand_dims(1 / tf.reduce_sum(weights, axis=1), axis=-1)
  weighted_imgs = tf.reshape(tf.reduce_sum(weighted_imgs, axis=1), [len(weights), 64*64*3])

  return weights * weighted_imgs

And my problematic Pytorch code:
def get_weighted_imgs(points, centers, imgs):
  weights = torch.Tensor([[torch.norm(p - c) for c in centers] for p in points])
  weighted_imgs = torch.Tensor([[w * img for w, img in zip(weight, imgs)] for weight in weights])

  weights = torch.unsqueeze(1 / torch.sum(weights, dim=1), dim=-1)
  weighted_imgs = torch.sum(weighted_imgs, dim=1).view([len(weights), 64*64*3])

  return weights * weighted_imgs

def reproducible():
  points = torch.Tensor(np.random.random((128, 5)))
  centers = torch.Tensor(np.random.random((10, 5)))
  imgs = torch.Tensor(np.random.random((10, 64, 64, 3)))

  weighted_imgs = get_weighted_imgs(points, centers, imgs)

I can guarantee that there is no issue with the dimension order or shape of the tensors/arrays. The error message I got is
ValueError: only one element tensors can be converted to Python scalars

which comes from
weighted_imgs = torch.Tensor([[w * img for w, img in zip(weight, imgs)] for weight in weights])

Could someone help me to solve this problem? That would be greatly appreciated.

Output:
Perhaps this will help you, but I'm not sure about your final multiplication between weights and weighted_imgs since they don't have the same shape, even after reshaping as you probably wanted. I am not sure I understood correctly your logic:
import torch
def get_weighted_imgs(points, centers, imgs):
  weights = torch.Tensor([[torch.norm(p - c) for c in centers] for p in points])

  imgs = imgs.unsqueeze(0).repeat(weights.shape[0],1,1,1,1)
  dims_to_rep = list(imgs.shape[-3:])
  weights = weights.unsqueeze(-1).unsqueeze(-1).unsqueeze(-1).repeat(1,1,*dims_to_rep)
  weights /= torch.sum(weights[...,0:1,0:1,0:1],dim=1, keepdim=True)
  weighted_imgs =  torch.sum(imgs * weights, dim=1).view(weights.shape[0], -1)

  return weighted_imgs #weights.view(weighted_imgs.shape[0],-1) *\
         #weighted_imgs # Shapes are torch.Size([128, 122880]) and torch.Size([128, 12288])

def reproducible():
  points = torch.Tensor(np.random.random((128, 5)))
  centers = torch.Tensor(np.random.random((10, 5)))
  imgs = torch.Tensor(np.random.random((10, 64, 64, 3)))

  weighted_imgs = get_weighted_imgs(points, centers, imgs)
#Test:
reproducible()


Instruction: Pytorch Unfold and Fold: How do I put this image tensor back together again?

Input:
I am trying to filter a single channel 2D image of size 256x256 using unfold to create 16x16 blocks with an overlap of 8. This is shown below:
# I = [256, 256] image
kernel_size = 16
stride = bx/2
patches = I.unfold(1, kernel_size,
int(stride)).unfold(0, kernel_size, int(stride)) # size = [31, 31, 16, 16]

I have started to attempt to put the image back together with fold but I’m not quite there yet. I’ve tried to use view to get the image to ‘fit’ the way it’s supposed to but I don’t see how this would preserve the original image. Perhaps I’m overthinking this.
# patches.shape = [31, 31, 16, 16]
patches = = filt_data_block.contiguous().view(-1, kernel_size*kernel_size) # [961, 256]
patches = patches.permute(1, 0) # size = [951, 256]

Any help would be greatly appreciated. Thanks very much.

Output:
A slightly less elegant solution than that proposed by Gil:
I took inspiration from this post on the Pytorch forums, formatting my image tensor to be of standard shape B x C x H x W (1 x 1 x 256 x 256). Unfolding:
# CREATE THE UNFOLDED IMAGE SLICES
I = image           # shape [256, 256]
kernel_size = bx    #shape [16]
stride = int(bx/2)  #shape [8]
I2 = I.unsqueeze(0).unsqueeze(0) #shape [1, 1, 256, 256]
patches2 = I2.unfold(2, kernel_size, stride).unfold(3, kernel_size, stride)
#shape [1, 1, 31, 31, 16, 16]

Following this, I do some transforms and filtering to my tensor stack. Before doing this I apply a cosine window and normalise:
# NORMALISE AND WINDOW
Pvv = torch.mean(torch.pow(win, 2))*torch.numel(win)*(noise_std**2)
Pvv = Pvv.double()
mean_patches = torch.mean(patches2, (4, 5), keepdim=True)
mean_patches = mean_patches.repeat(1, 1, 1, 1, 16, 16)
window_patches = win.unsqueeze(0).unsqueeze(0).unsqueeze(0).unsqueeze(0).repeat(1, 1, 31, 31, 1, 1)
zero_mean = patches2 - mean_patches
windowed_patches = zero_mean * window_patches

#SOME FILTERING ....

#ADD MEAN AND WINDOW BEFORE FOLDING BACK TOGETHER.
filt_data_block = (filt_data_block + mean_patches*window_patches) * window_patches

The above code works for me, but a mask would be more simple. Next, I prepare my tensor of form [1, 1, 31, 31, 16, 16] to be transformed back into the original [1, 1, 256, 256]:
# REASSEMBLE THE IMAGE USING FOLD
patches = filt_data_block.contiguous().view(1, 1, -1, kernel_size*kernel_size)
patches = patches.permute(0, 1, 3, 2)
patches = patches.contiguous().view(1, kernel_size*kernel_size, -1)
IR = F.fold(patches, output_size=(256, 256), kernel_size=kernel_size, stride=stride)
IR = IR.squeeze()

This allowed me to create an overlapping sliding window and seamlessly stitch the image back together. Cutting out the filtering makes for an identical image.


Instruction: What does Pytorch's nn.Linear(x,y) return?

Input:
I am new to object orientation, and I am having troubles understanding the following:
import torch.nn as nn

class mynet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(20, 64)

    def forward(self, x):
        x = self.fc1(x)

The line self.fc1 = nn.Linear(20, 64)  is supposed to create a member variable fc1 to my class, right? But what is the return value of nn.Linear(20, 64)?
According to the documentation, nn.Linear is defined as
class torch.nn.Linear(in_features: int, out_features: int, bias: bool = True).
However, in my basic OOP tutorial I have only seen something like class CLASSNAME(BASECLASS) so that the class CLASSNAME inherits from BASECLASS. What does the documentation mean with its way of writing all that stuff in between the brackets?
Also, the line x=fc1(x) somehow makes it look as if fc1 was a function now.
I seem to lack OOP knowledge here... Any help appreciated!

Output:
You can create a little examination:
import torch
import torch.nn as nnn
fc1 = nn.Linear(20, 64)
print(fc1, type(fc1))
ret = fc1(torch.randn(20))
print(ret, type(ret), ret.shape)

Out:
Linear(in_features=20, out_features=64, bias=True) <class 'torch.nn.modules.linear.Linear'>
tensor([-0.2795,  0.8476, -0.8207,  0.3943,  0.1464, -0.2174,  0.6605,  0.6072,
        -0.6881, -0.1118,  0.8226,  0.1515,  1.3658,  0.0814, -0.8751, -0.9587,
         0.1310,  0.2539, -0.3072, -0.0225,  0.4663, -0.0019,  0.0404,  0.9279,
         0.4948, -0.3420,  0.9061,  0.1752,  0.1809,  0.5917, -0.1010, -0.3210,
         1.1910,  0.5145,  0.2254,  0.2077, -0.0040, -0.6406, -0.1885,  0.5270,
         0.0824, -0.0787,  1.5140, -0.7958,  1.1727,  0.1862, -1.0700,  0.0431,
         0.6849,  0.1393,  0.7547,  0.0917, -0.3264, -0.2152, -0.0728, -0.6441,
        -0.1162,  0.4154,  0.3486, -0.1693,  0.6697,  0.0229,  0.0311,  0.1433],
       grad_fn=<AddBackward0>) <class 'torch.Tensor'> torch.Size([64])

fc1 is of type class 'torch.nn.modules.linear.Linear'.
It needs some "juice" to work. In your case it needs the input tensor torch.randn(20) to return the output of  torch.Size([64]).
So fc1 is a class instance that you can run with () in which case the forward() method of a class nn.Linear will be called.
In most cases when working with your modules (like mynet in your case) you will list the modules in __init__, and then in the forward of your module you will be defining what will happen (the behavior).
The three kind of modules in PyTorch are:

Functional modules
Default modules
Custom modules

Custom modules like mynet you created typically use default modules:

nn.Identity()
nn.Embedding()
nn.Linear()
nn.Conv2d()
nn.BatchNorm() (BxHxW)
nn.LayerNorm() (CxHxW)
nn.Dropout()
nn.ReLU()

And many many other modules that I haven't set. But of course, you can create custom modules without any default modules, just by using nn.Parameter(), see the last example.
The third kind functional modules are defined here.
Also check nn.Linear implementation. You may note the F.linear() functional module is used.
You may test the naive implementation of Linear from Fastai Book:
import torch
import torch.nn as nn
import math

class Linear(nn.Module):
    def __init__(self, n_in, n_out):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(n_out, n_in) * math.sqrt(2/n_in))
        self.bias = nn.Parameter(torch.zeros(n_out))

    def forward(self, x): return x @ self.weight.T + self.bias

fc = Linear(20,64)
ret = fc(torch.randn(20))
print(ret.shape) # 64

You may try to understand the difference between the naive implementation provided inside PyTorch.


Instruction: Install specific PyTorch version (pytorch==1.0.1)

Input:
I'm trying to install specific PyTorch version under conda env:
Using pip:
pip3 install pytorch==1.0.1
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement pytorch==1.0.1 (from versions: 0.1.2, 1.0.2)
ERROR: No matching distribution found for pytorch==1.0.1

Using Conda:
conda install pytorch==1.0.1
    Collecting package metadata (current_repodata.json): done
    Solving environment: failed with initial frozen solve. Retrying with flexible solve.
    Collecting package metadata (repodata.json): done
    Solving environment: failed with initial frozen solve. Retrying with flexible solve.

    PackagesNotFoundError: The following packages are not available from current channels:

      - pytorch==1.0.1

    Current channels:

      - https://repo.anaconda.com/pkgs/main/osx-64
      - https://repo.anaconda.com/pkgs/main/noarch
      - https://repo.anaconda.com/pkgs/r/osx-64
      - https://repo.anaconda.com/pkgs/r/noarch

    To search for alternate channels that may provide the conda package you're
    looking for, navigate to

        https://anaconda.org

    and use the search bar at the top of the page.

I was able to find this version under https://anaconda.org/soumith/pytorch but is there a way to find it and install from console?

Output:
You can download/install the version you like from the official Pytorch's Conda package. the link you specified is an old version and is not supported/updated for quit some time now!.
Install your desired version like this :
conda install pytorch==1.0.1 torchvision==0.2.2 -c pytorch

If you are looking for a pip version, you can view and access all versions from here as well.
and simply do :
pip install torch===1.0.1 -f https://download.pytorch.org/whl/torch_stable.html

You can always check the previous versions here as well.


Instruction: Pytorch reading tensors from file of tensors (stream training from disk)

Input:
I have some really big input tensors and I was running into memory issues while building them, so I read them one by one into a .pt file. As I run the script that generates and saves the file, the file gets bigger and bigger, so I am assuming that the tensors are saving correctly. Here is that code:
with open(a_sync_save, "ab") as f:
     print("saved")
     torch.save(torch.unsqueeze(torch.cat(tensors, dim=0), dim=0), f)

I want to read a certain amount of these tensors from the file at a time, because I do not want to run into a memory issue again. When I try to read each tensor saved to the file I can only manage to get the first tensor.
with open(a_sync_save, "rb") as f:
    for tensor in torch.load(f):
        print(tensor.shape)

The output here is the shape of the first tensor, then quits peacfully.

Output:
Here is some code that I used to answer this question. A lot of it is specific to what I am doing, but the jist of it can be used by others who are facing the same problem I was.
def stream_training(filepath, epochs=100):
    """
    :param filepath: file path of pkl file
    :param epochs: number of epochs to run
    """
    def training(train_dataloader, model_obj, criterion, optimizer):
        for j, data in enumerate(train_dataloader, start=0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data
            inputs, labels = inputs.cuda(), labels.cuda()
            outputs = model_obj(inputs.float())
            outputs = torch.flatten(outputs)
            loss = criterion(outputs, labels.float())
            print(loss)
            # zero the parameter gradients
            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model_obj.parameters(), max_norm=1)
            optimizer.step()

    tensors = []
    expected_values = []
    model= Model(1000, 1, 256, 1)
    model.cuda()
    criterion = nn.BCELoss()
    optimizer = optim.Adam(model.parameters(), lr=0.00001, betas=(0.9, 0.99999), eps=1e-08, weight_decay=0.001,
                           amsgrad=True)
    for i in range(epochs):
        with (open(filepath, 'rb')) as openfile:
            while True:
                try:
                    data_list = pickle.load(openfile)
                    tensors.append(data_list[0])
                    expected_values.append(data_list[1])
                    if len(tensors) % BATCH_SIZE == 0:
                        tensors = torch.cat(tensors, dim=0)
                        tensors = torch.reshape(tensors, (tensors.shape[0], tensors.shape[1], -1))
                        train_loader = make_dataset(tensors, expected_values) # makes a dataloader for the batch that comes in
                        training(train_loader, model, criterion, optimizer)  #Performs forward and back prop
                        tensors = [] # washes out the batch to conserve memory on my computer.
                        expected_values = []
                except EOFError:
                    print("This file has finished training")
                    break

Here is the model for fun.
class Model(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(Model, self).__init__()
        # dimensions
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers

        #Define the layers
        #GRU
        self.gru = nn.GRU(input_size, hidden_dim, n_layers, batch_first=True)
        self.fc1 = nn.Linear(hidden_dim, hidden_dim)
        self.bn1 = nn.BatchNorm1d(num_features=hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.bn2 = nn.BatchNorm1d(num_features=hidden_dim)
        self.fc3 = nn.Linear(hidden_dim, hidden_dim)
        self.bn3 = nn.BatchNorm1d(num_features=hidden_dim)
        self.fc4 = nn.Linear(hidden_dim, hidden_dim)
        self.bn4 = nn.BatchNorm1d(num_features=hidden_dim)
        self.fc5 = nn.Linear(hidden_dim, hidden_dim)
        self.output = nn.Linear(hidden_dim, output_size)

    def forward(self, x):
        x = x.float()
        x = F.relu(self.gru(x)[1])
        x = x[-1,:,:] # eliminates first dim
        x = F.dropout(x, 0.5)
        x = F.relu(self.bn1(self.fc1(x)))
        x = F.dropout(x, 0.5)
        x = F.relu(self.bn2(self.fc2(x)))
        x = F.dropout(x, 0.5)
        x = F.relu(self.bn3(self.fc3(x)))
        x = F.dropout(x, 0.5)
        x = F.relu(self.bn4(self.fc4(x)))
        x = F.dropout(x, 0.5)
        x = F.relu(self.fc5(x))
        return torch.sigmoid(self.output(x))

    def init_hidden(self, batch_size):
        hidden = torch.zeros(self.n_layers, batch_size, self.hidden_dim)
        return hidden


Instruction: Training custom model

Input:
I am trying to train my dataset on yolov5 I normalized data as discussed in the docs on github but I always end up with this error.
              from  n    params  module                                  arguments
  0             -1  1      8800  models.common.Focus                     [3, 80, 3]
  1             -1  1    115520  models.common.Conv                      [80, 160, 3, 2]
  2             -1  1    315680  models.common.BottleneckCSP             [160, 160, 4]
  3             -1  1    461440  models.common.Conv                      [160, 320, 3, 2]
  4             -1  1   3311680  models.common.BottleneckCSP             [320, 320, 12]
  5             -1  1   1844480  models.common.Conv                      [320, 640, 3, 2]
  6             -1  1  13228160  models.common.BottleneckCSP             [640, 640, 12]
  7             -1  1   7375360  models.common.Conv                      [640, 1280, 3, 2]
  8             -1  1   4099840  models.common.SPP                       [1280, 1280, [5, 9, 13]]
  9             -1  1  20087040  models.common.BottleneckCSP             [1280, 1280, 4, False]
 10             -1  1    820480  models.common.Conv                      [1280, 640, 1, 1]
 11             -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
 12        [-1, 6]  1         0  models.common.Concat                    [1]
 13             -1  1   5435520  models.common.BottleneckCSP             [1280, 640, 4, False]
 14             -1  1    205440  models.common.Conv                      [640, 320, 1, 1]
 15             -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
 16        [-1, 4]  1         0  models.common.Concat                    [1]
 17             -1  1   1360960  models.common.BottleneckCSP             [640, 320, 4, False]
 18             -1  1    922240  models.common.Conv                      [320, 320, 3, 2]
 19       [-1, 14]  1         0  models.common.Concat                    [1]
 20             -1  1   5025920  models.common.BottleneckCSP             [640, 640, 4, False]
 21             -1  1   3687680  models.common.Conv                      [640, 640, 3, 2]
 22       [-1, 10]  1         0  models.common.Concat                    [1]
 23             -1  1  20087040  models.common.BottleneckCSP             [1280, 1280, 4, False]
 24   [17, 20, 23]  1         0  models.yolo.Detect                      [3, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]]]
Traceback (most recent call last):
  File "train.py", line 404, in <module>
    train(hyp)
  File "train.py", line 80, in train
    model = Model(opt.cfg).to(device)
  File "/content/yolov5/models/yolo.py", line 62, in __init__
    m.stride = torch.tensor([128 / x.shape[-2] for x in self.forward(torch.zeros(1, ch, 128, 128))])  # forward
  File "/content/yolov5/models/yolo.py", line 90, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "/content/yolov5/models/yolo.py", line 107, in forward_once
    x = m(x)  # run
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/yolov5/models/yolo.py", line 26, in forward
    x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
RuntimeError: shape '[1, 3, 8, 16, 16]' is invalid for input of size 81920

These are the flags used
!python train.py --img 1024 --batch 4 --epochs 30 \
  --data ./data/mask.yaml --cfg ./models/yolov5x.yaml --weights yolov5x.pt \
   --cache --name maskmodel

this is the file structure

Output: For anyone who was facing the same problem, I found my issue when splitting data for training and validation make sure you pick a seed. Furthermore, when normalizing files for yolov5 input make sure that they ID as number without text in them. Thank you


Instruction: How To Import The MNIST Dataset From Local Directory Using PyTorch

Input:
I am writing a code of a well-known problem MNIST database of handwritten digits in PyTorch. I downloaded the train and testing dataset (from the main website) including the labeled dataset. The dataset format is t10k-images-idx3-ubyte.gz and after extract t10k-images-idx3-ubyte. My dataset folder looks like
MINST
 Data
  train-images-idx3-ubyte.gz
  train-labels-idx1-ubyte.gz
  t10k-images-idx3-ubyte.gz
  t10k-labels-idx1-ubyte.gz

Now, I wrote a code to load data like bellow
def load_dataset():
    data_path = "/home/MNIST/Data/"
    xy_trainPT = torchvision.datasets.ImageFolder(
        root=data_path, transform=torchvision.transforms.ToTensor()
    )
    train_loader = torch.utils.data.DataLoader(
        xy_trainPT, batch_size=64, num_workers=0, shuffle=True
    )
    return train_loader

My code is showing Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp
How can I solve this problem and I also want to check that my images are loaded (just a figure contains the first 5 images) from the dataset?

Output:
Read this Extract images from .idx3-ubyte file or GZIP via Python

Update
You can import data using this format
xy_trainPT = torchvision.datasets.MNIST(
    root="~/Handwritten_Deep_L/",
    train=True,
    download=True,
    transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor()]),
)

Now, what is happening at download=True first your code will check at the root directory (your given path) contains any datasets or not.
If no then datasets will be downloaded from the web.
If yes this path already contains a dataset then your code will work using the existing dataset and will not download from the internet.
You can check, first give a path without any dataset (data will be downloaded from the internet), and then give another path which already contains dataset data will not be downloaded.


Instruction: n_jobs > 1 using sklearn and pytorch is possible inside Neuraxle?

Input:
I built my own sklearn-like estimator using pytorch training inside GPU (cuda) and it works fine with RandomizedSearchCV when n_jobs==1. When n_jobs > 1, I get the following error:

PicklingError: Can't pickle <class 'main.LSTM'>: attribute lookup LSTM on main failed

This is the piece of code giving me the error:
model = my_model(input_size=1, hidden_layer_size=80, n_lstm_units=3, bidirectional=False,
                 output_size=1, training_batch_size=60, epochs=7500, device=device)
model.to(device)

hidden_layer_size = random.uniform(40, 200, 20).astype("int")
n_lstm_units = arange(1, 4)

parametros = {'hidden_layer_size': hidden_layer_size, 'n_lstm_units': n_lstm_units}

splitter = ShuffleSplit()

regressor = model
cv_search = \
    RandomizedSearchCV(estimator=regressor, cv=splitter,
                  search_spaces=parametros,
                  refit=True,
                  n_iter=4,
                  verbose=1,
                  n_jobs=2,
                  scoring=make_scorer(mean_squared_error,
                                      greater_is_better=False,
                                      needs_proba=False))

cv_search = MetaSKLearnWrapper(cv_search)
cv_search.fit(X, y)

Using Neuraxle wrapper leads to exactly same error, changes nothing.
I found closest solution here, but still don't know how to use RandomizedSearchCV within Neuraxle. It is a brand new project, so I couldn't find an answer on their docs or community examples. If anyone can give me an example or a good indication it will save my life. Thank you
Ps: Any way to run RandomizedSearchCV with my pytorch model on the gpu without Neuraxle also helps, I just need n_jobs>1.
Ps2: My model has a fit() method that creates and moves tensors to the gpu and works already tested.

Output:
There are multiple criteria that must be respected here for your code to work:

You need to use Neuraxle's RandomSearch instead of sklearn's random search for this to work. Use Neuraxle's base classes when possible.
Make sure that you use a Neuraxle BaseStep for your pytorch model, instead of a sklearn base classe.
Also, you should create your PyTorch code only in the setup() method or later. You can't create a PyTorch model in the __init__ of the BaseStep that contains pytorch code. You will want to read this page.
You will probably have to create a Saver for your BaseStep that contains PyTorch code if you want to serialize and then load your trained pipeline again. You can see how we created our TensorFlow Saver for our TensorFlow BaseStep and do something similar. Your saver will probably be much simpler than ours due to the more eager nature of PyTorch. For instance, you could have self.model inside your extension of the BaseStep class. The role of the saver would be to save and strip away this simple variable from self, and to be able to reload it whenever needed.

To sum up: you'd need to create two classes, and your two classes should look very similar our two TensorFlow step and saver classes here, to the exception that you PyTorch model is in a self.model variable of your step.
I'd be glad to see your implementation of your PyTorch base step and of your PyTorch saver!
You could then also even use the AutoML class (see AutoML example here) to save experiments in a Hyperparameter Repository as seen in the example.


Instruction: Is the gradient of the sum equal to the sum of the gradients for a neural network in pytorch?

Input:
Let's suppose I have the code below and I want to calculate the jacobian of L, which is the prediction made by a neural network in Pytorch, L is of size nx1 where n is the number of samples in a mini batch. In order to avoid a for loop for each entry of L (n entries) to calculate the jacobian for each sample in the mini batch some codes I found just sum the n predictions of the neural network (L) with respect with the inputs and then calculate the gradient of the sum. First I can't understand why is the gradient of the sum the same of the sum of the gradients for each sample in pytorch architecture. Second I tried both with the sum and with a for loop and the results diverge. Could it be due to numerical approximations or because the sum just doesn't make sense?
The code is below, where both functions belong to a nn.module:
def forward(self, x):
        with torch.set_grad_enabled(True):
            def function(x,t):
                 self.n = n = x.shape[1]//2

                 qqd = x.requires_grad_(True)
                 L = self._lagrangian(qqd).sum()
                 J = grad(L, qqd, create_graph=True)[0]

def _lagrangian(self, qqd):
    x = F.softplus(self.fc1(qqd))
    x = F.softplus(self.fc2(x))
    x = F.softplus(self.fc3(x))
    L = self.fc_last(x)
    return L

Output:
I think it should, this is just a toy example
w = torch.tensor([2.], requires_grad=True)
x1 = torch.tensor([3.], requires_grad=True)
x2 = torch.tensor([4.], requires_grad=True)
y = w * a + w * b
y.backward() # calculate gradient

return
>>> w.grad
tensor([7.])


Instruction: How to downgrade CUDA to 10.0.10 with conda, without conflicts?

Input:
I would like to go to CUDA (cudatoolkit) version compatible with
Nvidie-430 driver, i.e., 10.0.130 as recommended by the Nvidias site.
Based on this answer I did,
conda install -c pytorch cudatoolkit=10.0.130
And then I get this error (pastebin
link). (very-short version below):
(fastaiclean) eghx@eghx-nitro:~$ conda install -c pytorch cudatoolkit=10.0.130
...
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package _libgcc_mutex conflicts for:
pyzmq -> libgcc-ng[version='>=7.3.0'] -> _libgcc_mutex=[build=main]
libgcc-ng -> _libgcc_mutex=[build=main]
lcms2 -> libgcc-ng[version='>=7.3.0'] -> _libgcc_mutex=[build=main]
...
The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__cuda==10.1=0
  - feature:|@/linux-64::__cuda==10.1=0

Your installed version is: 10.1

Why am I getting conflicts? Why does it say 10.1 when cuda toolkit
is 10.2.89 (conda list)? how to handle conflicts? What can I do
with this error? The conflicts are so huge, I don't know where to start.
Other

Nvidia driver 430

current cudatoolkit: 10.2.89

Output:
Check current version with
torch.version.cuda

I had 10.2. But I need 10.1 according to: table 1 here and my 430 NVIDIA driver installed.
Uninstall and Install
conda remove pytorch torchvision cudatoolkit

conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1.168 -c pytorch

Say yes to everything for the above commands.


Instruction: Pytorch detection of CUDA

Input:
Which is the command to see the "correct" CUDA Version that pytorch in conda env is seeing?  This, is a similar question, but doesn't get me far.

nvidia-smi says I have cuda version 10.1

conda list tells me cudatoolkit version is 10.2.89

torch.cuda.is_available() shows FALSE, so it sees No CUDA?

print(torch.cuda.current_device()), I get 10.0.10 (10010??) (it
looks like):

AssertionError: The NVIDIA driver on your system is too old
(found version 10010)

print(torch._C._cuda_getCompiledVersion(), 'cuda compiled version') tells me my version is 10.0.20 (10020??)?

10020 cuda compiled version

Why are there so many different versions? What am I missing?
P.S
I have Nvidia driver 430 on Ubuntu 16.04 with Geforce 1050. It comes
with libcuda1-430 when I installed the driver from additional drivers tab in ubuntu (Software and Updates). I installed pytorch
with conda which also installed the cudatoolkit using conda install -c fastai -c pytorch -c anaconda fastai

Output:
In the conda env (myenv) where pytorch is installed do the following:
conda activate myenv

torch.version.cuda

Nvidia-smi only shows compatible version. Does not seem to talk about the version pytorch's own cuda is built on.


Instruction: Validation dataset in PyTorch using DataLoaders

Input:
I want to load MNIST dataset in PyTorch and Torchvision, dividing it into train, validation and test parts. So far I have:
def load_dataset():
    train_loader = torch.utils.data.DataLoader(
        torchvision.datasets.MNIST(
            '/data/', train=True, download=True,
            transform=torchvision.transforms.Compose([
                torchvision.transforms.ToTensor()])),
        batch_size=batch_size_train, shuffle=True)

    test_loader = torch.utils.data.DataLoader(
        torchvision.datasets.MNIST(
            '/data/', train=False, download=True,
            transform=torchvision.transforms.Compose([
                torchvision.transforms.ToTensor()])),
        batch_size=batch_size_test, shuffle=True)

How can I divide the training dataset into training and validation if it's in the DataLoader? I want to use last 10000 examples from the training dataset as a validation dataset (I know that I should do CV for more accurate results, I just want a quick validation here).

Output:
Splitting the training dataset into training and validation in PyTorch turns out to be much harder than it should be.
First, split the training set into training and validation subsets (class Subset), which are not datasets (class Dataset):
train_subset, val_subset = torch.utils.data.random_split(
        train, [50000, 10000], generator=torch.Generator().manual_seed(1))

Then get actual data from those datasets:
X_train = train_subset.dataset.data[train_subset.indices]
y_train = train_subset.dataset.targets[train_subset.indices]

X_val = val_subset.dataset.data[val_subset.indices]
y_val = val_subset.dataset.targets[val_subset.indices]

Note that this way we don't have Dataset objects, so we can't use DataLoader objects for batch training. If you want to use DataLoaders, they work directly with Subsets:
train_loader = DataLoader(dataset=train_subset, shuffle=True, batch_size=BATCH_SIZE)
val_loader = DataLoader(dataset=val_subset, shuffle=False, batch_size=BATCH_SIZE)


Instruction: Computing mean of list of generator objects

Input:
// Initialize 2-D array, each entry is some neural network
phi = [[None] * n for _ in range(m)]
for i in range(m):
    for j in range(n):
        phi[i][j] = NeuralNetwork()

// Let k, i be arbitrary indices
p1 = torch.nn.utils.parameters_to_vector(phi[k][i - 1].parameters())
p2 = torch.nn.utils.parameters_to_vector(mean of phi[:][i-1])

I want to basically compute the mean squared error between the parameters phi[k][i-1] and average of the entire column phi[:][i-1] i.e. ((p1 - p2)**2).sum() I tried in the following way:
tmp = [x.parameters() for x in self.phi[:][i - 1]]
mean_params = torch.mean(torch.stack(tmp), dim=0)
p2 = torch.nn.utils.parameters_to_vector(mean_params)

But this doesn't work out because tmp is a list of generator objects. More specifically, I guess my problem is to compute the mean from that generator object.

Output:
First we can define a function that computes the average parameters for a list of models. To avoid creating a copy of the parameters of each model all at the same time we probably want to compute this as a running sum. For example
def average_parameters_vector(model_list):
    n = len(model_list)
    avg = 0
    for model in model_list:
        avg = avg + torch.nn.utils.parameters_to_vector(model.parameters()) / n
    return avg

Then you can just create p1 and p2 and compute the mean-squared error
p1 = torch.nn.utils.parameters_to_vector(phi[k][i - 1].parameters())
p2 = average_parameters_vector(phi[:][i - 1])

mse = ((p1 - p2)**2).mean()

If you really want a one-line solution that's also possibly the fastest you could compute this by making a single tensor containing all the parameters of the models in phi[:][i - 1], then mean reducing them. But as mentioned earlier this will significantly increase memory usage, especially if your models have millions of parameters as is often the case.
# Uses lots of memory but potentially the fastest solution
def average_parameters_vector(model_list):
    return torch.stack([torch.nn.utils.parameters_to_vector(model.parameters()) for model in model_list]).mean(dim=0)

On the other extreme, if you're very concerned about memory usage then you could compute the average of each individual parameter at a time.
# more memory efficient than original solution but probably slower
def average_parameters_vector(model_list):
    n = len(model_list)
    num_params = len(list(model_list[0].parameters()))
    averages = [0] * num_params
    for model in model_list:
        for pidx, p in enumerate(model.parameters()):
            averages[pidx] = averages[pidx] + p.data.flatten() / n
    return torch.cat(averages)


Instruction: Shouldn't same neural network weights produce same results?

Input:
So I am working with different deep learning frameworks as part of my research and have observed something weird (at least I cannot explain the cause of it).
I trained a fairly simple MLP model (on mnist dataset) in Tensorflow, extracted trained weights, created the same model architecture in PyTorch and applied the trained weights to PyTorch model. Now my expectation is to get same test accuracy from both Tensorflow and PyTorch models but this isn't the case. I get different results.
So my question is: If a model is trained to some optimal value, shouldn't the trained weights produce same results every time testing is done on the same dataset (regardless of the framework used)?
PyTorch Model:
class Net(nn.Module):

def __init__(self) -> None:
    super(Net, self).__init__()
    self.fc1 = nn.Linear(784, 24)
    self.fc2 = nn.Linear(24, 10)

def forward(self, x: Tensor) -> Tensor:
    x = torch.flatten(x, 1)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

Tensorflow Model:
def build_model() -> tf.keras.Model:
    # Build model layers
    model = models.Sequential()
    # Flatten Layer
    model.add(layers.Flatten(input_shape=(28,28)))
    # Fully connected layer
    model.add(layers.Dense(24, activation='relu'))
    model.add(layers.Dense(10))
    # compile the model
    model.compile(
        optimizer='sgd',
        loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=['accuracy']
    )
    # return newly built model
    return model

To extract weights from Tensorflow model and apply them to Pytorch model I use following functions:
Extract Weights:
def get_weights(model):
    # fetch latest weights
    weights = model.get_weights()
    # transpose weights
    t_weights = []
    for w in weights:
        t_weights.append(np.transpose(w))
    # return
    return t_weights

Apply Weights:
def set_weights(model, weights):
    """Set model weights from a list of NumPy ndarrays."""
    state_dict = OrderedDict(
        {k: torch.Tensor(v) for k, v in zip(model.state_dict().keys(), weights)}
    )
    self.load_state_dict(state_dict, strict=True)

Output:
Providing solution in answer section for the benefit of community. From comments

If you are using the same weights in the same manner then results
should be the same, though float rounding error should also be
accounted. Also it doesn't matter if model is trained at all. You can
think of your model architecture as a chain of matrix multiplications
with element-wise nonlinearities in between. How big is the
difference? Are you comparing model outputs, our metrics computed over
dataset? As a suggestion, intialize model with some random values in
Keras, do a forward pass for a single batch (paraphrased from jdehesa and Taras Sereda)


Instruction: Predict only one class (person) in YOLACT/YOLACT++

Input:
I want to predict only one class i.e. person from all the 84 classes that are being checked for and predicted.
For YOLACT reference https://github.com/dbolya/yolact
The results are pretty fine but I guess I just need to modify one of the codes and in a very short way but I cant manage to find out
There is one issue related to this in which I did what he mentioned like adding the 4 lines in Yolact/layers/output_utils.py and changing nothing else. Those lines are as following:
boxes = torch.cat((boxes[classes==0], boxes[classes==2]),dim=0)
scores = torch.cat((scores[classes==0], scores[classes==2]),dim=0)
masks = torch.cat((masks[classes==0], masks[classes==2]),dim=0)
classes = torch.cat((classes[classes==0], classes[classes==2]),dim=0)

But it gives the following error:
RuntimeError: strides[cur - 1] == sizes[cur] * strides[cur] INTERNAL ASSERT FAILED at
/opt/conda/conda-bld/pytorch_1573049310284/work/torch/csrc/jit/fuser/executor.cpp:175,
please report a bug to PyTorch.
The above operation failed in interpreter, with the following stack trace:

terminate called without an active exception
Aborted (core dumped)

I tried adding the if condition as mentioned but still it gives error. I am using pytorch 1.3

Output:
In order to show a single class (person, id:0) output at the time of inference, you simply need to add
cur_scores[1:] *= 0

after cur_scores = conf_preds[batch_idx, 1:, :] in line 83 of yolact/layers/functions/detection.py.
Then running
!python eval.py --trained_model=weights/yolact_resnet50_54_800000.pth --score_threshold=0.15 --top_k=15 --image=input_image.png:output_image.png

will give you single class inference.
As mentioned by the author in issue#218:

you can make the change to save on NMS computation,
simply add cur_scores[<everything but your desired class>] *= 0
For the index, if you wanted only person (class 0), you can put 1:, but if you wanted another class than that you'd need to do 2 statements: one with :<class_idx> and the other with <class_idx>+1:. Then when you run eval, run it with --cross_class_nms=True and that'll remove all the other classes from NMS.

Other method is to modify the output in output_utils.py.


Instruction: How to convert a list of Torch tensor with grad to tensor

Input:
I have a variable called pts which is shaped [batch, ch, h, w]. This is a heatmap and I want to convert it to 2nd co-ordinates. The goal is, pts_o = heatmap_to_pts(pts) where pts_o will be [batch, ch, 2]. I have wrote this function so far,
def heatmap_to_pts(self, pts):  <- pts [batch, 68, 128, 128]

    pt_num = []

    for i in range(len(pts)):

        pt = pts[i]
        if type(pt) == torch.Tensor:

            d = torch.tensor(128)                                                   * get the
            m = pt.view(68, -1).argmax(1)                                           * indices
            indices = torch.cat(((m / d).view(-1, 1), (m % d).view(-1, 1)), dim=1)  * from heatmaps

            pt_num.append(indices.type(torch.DoubleTensor) )   <- store the indices in a list

    b = torch.Tensor(68, 2)                   * trying to convert
    c = torch.cat(pt_num, out=b) *error*      * a list of tensors with grad
    c = c.reshape(68,2)                       * to a tensor like [batch, 68, 2]

    return c

The error says "cat(): functions with out=... arguments don't support automatic differentiation, but one of the arguments requires grad.". It's unable to do the operations because tensors in pt_num requires grad".
How can I convert that list to a tensor?

Output:
The error says,

cat(): functions with out=... arguments don't support automatic differentiation, but one of the arguments requires grad.

What that means is that the output of functions such as torch.cat() which as an out= kwarg cannot be used as input to the autograd engine (which performs automatic differentiation).
The reason is that the tensors (in your Python list pt_num) have different values for the requires_grad attribute, i.e., some tensors have requires_grad=True while some of them have requires_grad=False.
In your code, the following line is (logically) troublesome:
c = torch.cat(pt_num, out=b)

The return value of torch.cat(), irrespective of whether you use out= kwarg or not, is the concatenation of tensors along the mentioned dimension.
So, the tensor c is already the concatenated version of individual tensors in pt_num. Using out=b redundant. Thus, you can simply get rid of the out=b and everything should be fine.
c = torch.cat(pt_num)


Instruction: How to understand tensors with multiple dimensions?

Input:
I've been recently a bit confused about tensors. Say if we have a tensor with shape (3,2,3,4), we mean that in the first dimension there are 3 groups of numbers? OR it just means that there are exactly just 3 numbers in the first dimension?
Then here comes the second question, with the tensor A that has the shape (3,2), why is that the output of torch.max(A,0) returns a group of max values that contains 2 max values instead of 3,considering the fact that there are 3 numbers in the first dimension.
>>>a = torch.randn(3,2)
a tensor([[-1.1254, -0.1549],
        [-0.5308,  1.0427],
        [-0.1268,  1.0866]])
>>>torch.max(a,0)
torch.return_types.max(
values=tensor([-0.1268,  1.0866]),
indices=tensor([2, 2]))

I mean why doesn't it return a list of 3 max values?
Then the third question, if we have two tensors with shape(3,3,10,2) and (2,4,10,1), can we just concatenate these two tensors on the third dimension considering they have the same size on that dimension? If it is feasible, what is the reason behind it?
I'll be much appreciated if you help me understand this!

Output:
Tensors are just high level dimensions of vectors. You should start with a vector first. Ex: 4 elements vector A = [1,2,3,4]. A 6-vectors A together is B = [[1,2,3,4],[1,2,3,4],...,[1,2,3,4]]. This is called matrix (2 dimensions), the shape is 6x4. Now if you take 2 matrices B, it will form a tensor C, shape is (2,6,4). Extend C for a list of 3 C, you can get a 4-dimensions tensor (3,2,6,4) and so on. You can take this picture for better illustration.

For the torch max, torch.max(input, dim, keepdim=False, out=None) when you choose the dim=0 (the dimension to reduce to  this mean you will find the max along the 0 axis (aka the first shape), it will follow through this axis to find the maximum number corresponding to the rest tensor (according to your tensor, you only have to check in the 2 elements vectors), which is left. If you extend this for higher dimension, like (3,4,5) shape, the max will be (4,5).
For the last question, there is a no for that tensor. Because when you look at the illustration figure, there is no way for you to just base on 1 shape to concatenate 2 different size matching tensor. You have to maintain the same shape for all dimensions except the on you are going to concatenate.
>>> b = np.random.randint(0,2,size=(2,3,2,3))
>>> a = np.random.randint(0,2,size=(2,3,1,3))
>>> np.concatenate([a,b],axis=2)

Else, the only way for you to do it is flattening both of the 2 tensors so that you only need a vector represent for both a and b.


Instruction: How can we check if a matrix is PSD is PyTorch?

Input: There is a poste on checking if a matrix is PSD in Python. I am wondering how we can check it in PyTorch? is there a function for that?

Output:
Haven't found a PyTorch function for that, but you should be able to determine it easily, and similarly to the post you've linked, by checking whether the matrix is symmetric and all eigenvalues are non-negative:
def is_psd(mat):
    return bool((mat == mat.T).all() and (torch.eig(mat)[0][:,0]>=0).all())
#Test:
is_psd(torch.randn(2,2))


Instruction: PyTorch LSTM categorical model - output to target mapping

Input:
I have a network which outputs a vector of length two. My targets are in the form of 1 or zeros, referring to two possible categories. What is the best way to get the loss - i.e. should I transform the targets, for example into a dimension 2 vector, or should I transform the output of the network, e.g. take the location of the max number as the output?
My network looks like:
class LSTMClassifier(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm1 = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.lstm2 = nn.LSTM(hidden_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc1 = nn.Linear(hidden_dim, 32)
        self.fc2 = nn.Linear(32, 1)
        self.dropout = nn.Dropout(p=0.2)
        self.batch_normalisation1 = nn.BatchNorm1d(layer_dim)
        self.batch_normalisation2 = nn.BatchNorm1d(2)
        self.activation = nn.Softmax(dim=2)

    def forward(self, x):
        h0, c0 = self.init_hidden(x)
        out, (hn1, cn1) = self.lstm1(x, (h0, c0))
        out = self.dropout(out,)
        out = self.batch_normalisation1(out)

        h1, c1 = self.init_hidden(out)
        out, (hn2, cn2) = self.lstm2(out, (h1, c1))
        out = self.dropout(out)
        out = self.batch_normalisation1(out)

        h2, c2 = self.init_hidden(out)
        out, (hn3, cn3) = self.lstm2(out, (h2, c2))
        out = self.dropout(out)
        out = self.batch_normalisation1(out)

        out = self.fc1(out[:, -1, :])
        out = self.dropout(out)
        out = self.fc2(out)
        return out

    def init_hidden(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        return [t for t in (h0, c0)]

    def pred(self, x):
        out = self(x)
        return out > 0

An example of input to this network is:
tensor([[[0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],
         [2.3597e-04, 1.1507e-02, 8.7719e-02, 6.1093e-02, 9.5556e-01],
         [2.1474e-03, 5.3805e-03, 9.6491e-02, 2.2508e-01, 8.2222e-01]]])

which has shape torch.Size([1, 3, 5]). The target is currently 1 or 0. However, the network outputs a vector such as:
tensor([[0.5293, 0.4707]], grad_fn=<SoftmaxBackward>)

What would be the best way to set up the loss between these target and the network output?
Update:
I can now train the model as suggested in the answers as:
model = LSTMClassifier(5, 128, 3, 1)
Epochs = 10
batch_size = 32

criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=1e-6)

for epoch in range(Epochs):
    if epoch == 0:
        accurate = 0
        for X_instance, y_instance in zip(val_x, val_y):
            if int(y_instance) == 1 and model.pred(X_instance.view(-1, 3, 5)).item():
                accurate += 1
        print(f"Untrained accuracy test set: {accurate/len(val_x)}")
    print(f"Epoch {epoch + 1}")

    for n, (X, y) in enumerate(train_batches):
        model.train()
        optimizer.zero_grad()
        y_pred = model(X)
        loss = criterion(y_pred, y)
        loss.backward()
        optimizer.step()

    model.eval()
    accurate = 0
    for X_instance, y_instance in zip(val_x, val_y):
        if int(y_instance) == 1 and model.pred(X_instance.view(-1, 3, 5)).item():
            accurate += 1
    print(f"Accuracy test set: {accurate/len(val_x)}")

Output:
You shouldn't use any activation at the end of your network and output only a single neuron instead of two (trained with BCEWithLogitsLoss).
Below is your neural network code with commentary and removal of unnecessary parts:
class LSTMClassifier(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm1 = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.lstm2 = nn.LSTM(hidden_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc1 = nn.Linear(hidden_dim, 32)
        # Output 1 neuron instead of two
        self.fc2 = nn.Linear(32, 1)
        # Model should not depend on batch size
        # self.batch_size = None
        # You are not using this variable
        # self.hidden = None
        self.dropout = nn.Dropout(p=0.2)
        self.batch_normalisation1 = nn.BatchNorm1d(layer_dim)
        self.batch_normalisation2 = nn.BatchNorm1d(2)

    def forward(self, x):
        # Hidden are initialized with 0 explicitly
        # h0, c0 = self.init_hidden(x)
        out, _ = self.lstm1(x)
        # No need for initial values
        # out, (hn1, cn1) = self.lstm1(x, (h0, c0))
        out = self.dropout(out)
        out = self.batch_normalisation1(out)

        # Same for all other cells you re-init with zeros, it's implicit
        out, _ = self.lstm2(out)
        out = self.dropout(out)
        out = self.batch_normalisation1(out)

        out, _ = self.lstm2(out)
        out = self.dropout(out)
        out = self.batch_normalisation1(out)

        out = self.fc1(out[:, -1, :])
        out = self.dropout(out)
        # No need for activation
        # out = F.softmax(self.fc2(out))
        out = self.fc2(out)
        return out

    # Return True (1) or False (0)
    def pred(self, x):
        return self(x) > 0

I have also added pred method which transforms logits into targets (e.g. to use with some metrics).
Basically, if your logit is lower than 0 it is False, otherwise it is True. No need for activation in this case.