Instruction: Loading a single graph into a pytorch geometric data object for node classification

Input:
I have one graph, defined by 4 matrices: x (node features), y (node labels), edge_index (edges list) and edge_attr (edge features). I want to create a dataset in Pytorch Geometric with this single graph and perform node-level classification. It seems that just wrapping these 4 matrices into a data object fails, for some reason.
I have created a dataset containing the attributes:
Data(edge_attr=[3339730, 1], edge_index=[2, 3339730], x=[6911, 50000], y=[6911, 1])

representing a graph. If I try to slice this graph, like:
train_dataset, test_dataset = dataset[:5000], dataset[5000:]

I get the error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-feb278180c99> in <module>
      3 # train_dataset, test_dataset = torch.utils.data.random_split(dataset, [train_size, test_size])
      4
----> 5 train_dataset, test_dataset = dataset[:5000], dataset[5000:]
      6
      7 # Create dataloader for training and test dataset.

~/anaconda3/envs/py38/lib/python3.8/site-packages/torch_geometric/data/data.py in __getitem__(self, key)
     92     def __getitem__(self, key):
     93         r"""Gets the data of the attribute :obj:`key`."""
---> 94         return getattr(self, key, None)
     95
     96     def __setitem__(self, key, value):

TypeError: getattr(): attribute name must be string

What am I doing wrong in the data construction?

Output:
For node classification:
Create custom dataset.
class CustomDataset(InMemoryDataset):
    def __init__(self, root, transform=None, pre_transform=None):
        super(CustomDataset, self).__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self):
        return ['edge_list.csv', 'x.pt', 'y.pt', 'edge_attributes.csv']

    @property
    def processed_file_names(self):
        return ['graph.pt']

    def process(self):
        data_list = []
        edge_list = pd.read_csv(self.raw_paths[0], dtype=int)
        target_nodes = edge_list.iloc[:,0].values
        source_nodes = edge_list.iloc[:,1].values
        edge_index = torch.tensor([source_nodes, target_nodes], dtype=torch.int64)

        x = torch.load(self.raw_paths[1], map_location=torch.device('cpu'))
        y = torch.load(self.raw_paths[2], map_location=torch.device('cpu'))

        # make masks
        n = x.shape[0]
        randomassort = list(range(n))
        random.shuffle(randomassort)
        max_train = floor(len(randomassort) * .1)
        train_mask_idx = torch.tensor(randomassort[:max_train])
        test_mask_idx = torch.tensor(randomassort[max_train:])
        train_mask = torch.zeros(n); test_mask = torch.zeros(n)
        train_mask.scatter_(0, train_mask_idx, 1)
        test_mask.scatter_(0, test_mask_idx, 1)
        train_mask = train_mask.type(torch.bool)
        test_mask = test_mask.type(torch.bool)

        edge_attributes = pd.read_csv(self.raw_paths[3])

        data = Data(edge_index=edge_index, x=x, y=y, train_mask=train_mask, test_mask=test_mask)

        print(data.__dict__)
        data, slices = self.collate([data])
        torch.save((data, slices), self.processed_paths[0])

Then in the train loop use the masks when updating the model.
def train():
    ...
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()


Instruction: Pytorch BatchNorm3d / InstanceNorm3d not working when data size (1,C,1,1,1)

Input:
I'm training a neural network in PyTorch which at some point has a BatchNorm3d(C).
Normally, I'm training it with a batch size of 1, and the input of this specific level will then be of shape (1, C, 1, 1, 1).
Unfortunately, the BatchNorm then fails with the error message:

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 32, 1, 1, 1])

The same happens, when I use the InstanceNorm3d. It works perfectly fine when I use a batch size greater than two (i.e. the input will then be of shape (2, C, 1, 1, 1).
Does anyone know a solution to this problem? What am I missing?
The problem can be reproduced with the following snippet:
import torch
x_working = torch.ones([2,32,1,1,1])
x_not_working = torch.ones([1,32,1,1,1])
norm = torch.nn.InstanceNorm3d(32)
out_working = norm(x_working)
out_not_working=norm(x_not_working)

Output:
All these normalization layers keep running estimates of the mean and variance of your data over the batch dimension (see doc). Since the variance is computed with the unbiased estimator (notice the n-1 in the denominator), the computation cannot work with less than 2 data points. Therfore, you need a batch size of at least 2 to use these layers.
Note that the variance of 1 data point - if pytorch agreed to compute it - would always be 0, so not really interesting a result. Actually, Batchnorm is known to require significantly larger batch sizes (you often finds 32 or 64 in the scientific literature) to work properly. This strong requirement lead to the development of layerNorm and groupNorm (of which instanceNorm is a particular case I believe).


Instruction: Is it possible to save Tensorboad session?

Input:
I'm using Tensorboard and would like to save and send my report (email outside my organization), without losing interactive abilities. I've tried to save it as a complete html but that didn't work.
Anyone encountered the same issue and found a solution?

Output:
Have you seen tensorboard.dev?
This page allows you to host your tensorboard experiment & share it with others using a link (it's still interactive) for free.
Also you can use it from the command line; try this from your CLI for more information:
$ tensorboard dev --help


Instruction: TypeError: 'torch.dtype' object is not callable. how to call this function?

Input:
How to call this torch.dtype? because here the error shows it's not callable. before I used floatTensor and it shows the error like this can't convert np.ndarray of type numpy.object_ and now I using float64 it showing the error 'torch.dtype' object is not callable. please help with this issue.
import torch
    a = torch.cuda.is_available()
    b = torch.cuda.current_device()
    c = torch.cuda.get_device_name()
    d = torch.cuda.memory_reserved()
    e = torch.cuda.memory_allocated()
    var1 = torch.FloatTensor([1.0,2.0,3.0]).cuda()
    var1
    a1 = var1.device
    import pandas as pd
    df = pd.read_csv('diabetes.csv')
    df.head()
    b1 = df.isnull().sum()
    import seaborn as sns
    import numpy as np
    df['Outcome']=np.where(df['Outcome']==1,"Diabetic","No Diabetic")
    b2 = df.head()
    b3 = sns.pairplot(df,hue="Outcome")

    X=df.drop('Outcome',axis=1).values
    y=df['Outcome'].values

    from sklearn.model_selection import train_test_split

    X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)
    y_train

    import torch
    import torch.nn as nn
    import torch.nn.functional as F

    X_train=torch.FloatTensor(X_train).cuda()
    X_test=torch.FloatTensor(X_test).cuda()
    y_train=torch.float64(y_train).cuda()

this is the error:
 C:\Users\vinot\.conda\envs\python21\python.exe D:/python/python_work/pythonProject/diabetes.py
    Traceback (most recent call last):
      File "D:/python/python_work/pythonProject/diabetes.py", line 35, in <module>
        y_train=torch.float64(y_train).cuda()
    TypeError: 'torch.dtype' object is not callable

    Process finished with exit code 1

Output:
torch.float64 is a dtype object and not a function so it cannot be called.
To make it into a double float (or at least to make sure it is), I would instead call:
y_train = torch.from_numpy(y_train).double().cuda()


Instruction: How to get logits as neural network output

Input:
Simple and short question. I have a network (Unet) which performs image segmentation. I want the logits as the output to feed into the cross entropy loss (using pytorch). Currently my final layer looks as so:
class Logits(nn.Sequential):
    def __init__(self,
                 in_channels,
                 n_class
                 ):
      super(Logits, self).__init__()

      # fully connected layer outputting the prediction layers for each of my classes
      self.conv = self.add_module('conv_out',
                        nn.Conv2d(in_channels,
                                  n_class,
                                  kernel_size = 1
                                  )
                       )
      self.activ = self.add_module('sigmoid_out',
                        nn.Sigmoid()
                       )

Is it correct to use the sigmoid activation function here? Does this give me logits?

Output:
When people talk about "logits" they usually refer to the "raw" n_class-dimensional output vector. For multi-class classification (n_class > 2) you want to convert the n_class-dimensional vector of raw "logits" into a n_class-dim probability vector.
That is, you want prob = f(logits) with prob_i >= 0 for all n_class entries, and that sum(prob)=1.
The most straight forward way of doing that in a differentiable way is to use the Softmax function:
prob_i = softmax(logits) = exp(logits_i) / sum_j exp(logits_j)

It is easy to see that the output of softmax is indeed a n_class-dim probability vector (I leave it to you as a short exercise).
BTW, this is why the raw predictions are called "logits" because they are kind of "log" of the output predicted probabilities.
Now, it is customary not to explicitly compute the softmax on top of a classification network and defer its computation to the loss function, e.g. nn.CrossEntropyLoss that internally computes the softmax and requires the raw logits as inputs, rather than the normalized probabilities. This is done mainly for numerical stability.
Therefore, if you are training a multi-class classification network with nn.CrossEntropyLoss you do not need to worry at all about the final activation and simply output the raw logits from your final conv/linear layer.
Most importantly, do not use nn.Sigmoid() activation as it tends to have saturated gradients and will mess up your training.


Instruction: Using tensordot with torch.sparse tensors

Input:
Is it possible to use a similar method as "tensordot" with torch.sparse tensors?
I am trying to apply a 4 dimensional tensor onto a 2 dimensional tensor. This is possible using torch or numpy. However, I did not find the way to do it using torch.sparse without making the sparse tensor dense using ".to_dense()".
More precisely, here is what I want to do without using ".to_dense()":
import torch
import torch.sparse

nb_x = 4
nb_y = 3
coordinates = torch.LongTensor([[0,1,2],[0,1,2],[0,1,2],[0,1,2]])
values = torch.FloatTensor([1,2,3])
tensor4D = torch.sparse.FloatTensor(coordinates,values,torch.Size([nb_x,nb_y,nb_x,nb_y]))
inp = torch.rand((nb_x,nb_y))

#what I want to do
out = torch.tensordot(tensor4D.to_dense(),inp,dims=([2,3],[0,1]))

print(inp)
print(out)

(here is the output: torch_code)
Alternatively, here is a similar code using numpy:
import numpy as np

tensor4D = np.zeros((4,3,4,3))
tensor4D[0,0,0,0] = 1
tensor4D[1,1,1,1] = 2
tensor4D[2,2,2,2] = 3
inp = np.random.rand(4,3)

out = np.tensordot(tensor4D,inp)

print(inp)
print(out)

(here is the output: numpy_code)
Thanks for helping!

Output:
Your specific tensordot can be cast to a simple matrix multiplication by "squeezing" the first two and last two dimensions of tensor4D.
In short, what you want to do is
raw = tensor4D.view(nb_x*nb_y, nb_x*nb_y) @ inp.flatten()
out = raw.view(nb_x, nb_y)

However, since view and reshape are not implemented for sparse tensors, you'll have to it manually:
sz = tensor4D.shape
coeff = torch.tensor([[1, sz[1], 0, 0], [0, 0, 1, sz[3]]])
reshaped = torch.sparse.FloatTensor(coeff @ idx, tensor4D._values(), torch.Size([nb_x*nb_y, nb_x*nb_y]))

# once we reshaped tensord4D it's all downhill from here
raw = torch.sparse.mm(reshaped, inp.flatten()[:, None])
out = raw.reshape(nb_x, nb_y)
print(out)

And the output is

tensor([[0.4180, 0.0000, 0.0000],
   [0.0000, 0.6025, 0.0000],
   [0.0000, 0.0000, 0.5897],
   [0.0000, 0.0000, 0.0000]])


Instruction: Pytorch's Autograd does not support complex matrix inversion, does anyone have a workaround?

Input:
Somewhere in my loss function, I invert a complex matrix of size 64*64. Although complex matrix inversion is supported for torch.tensor, the gradient cannot be computed in the training loop as I get this error:
RuntimeError: inverse does not support automatic differentiation for outputs with complex type.
Does anyone have a workaround for this issue? a custom function instead of torch.inverse maybe?

Output:
You can do the inverse yourself using the real-valued components of your complex matrix.
Some linear algebra first:
a complex matrix C can be written as a sum of two real matrices A and B (j is the sqrt of -1):
C = A + jB

Finding the inverse of C is basically finding two real valued matrices x and y such that
(A + jB)(x + jy) = I + j0

This boils down to solving the real valued system of equations:

Now that we know how to do reduce a complex matrix inversion to real-valued matrix inversion, we can use pytorch's solve to do the inverse for us.
def complex_inverse(C):
  A = torch.real(C)
  B = torch.imag(C)
  # construct the left hand side of the system of equations
  # side note: from pytorch 1.7.1 you can use vstack and hstack instead of cat
  lhs = torch.cat([torch.cat([A, -B], dim=1), torch.cat([B, A], dim=1)], dim=0)
  # construct the rhs of the system of equations
  rhs = torch.cat([torch.eye(A.shape[0]).to(A), torch.zeros_like(A)],dim=0)

  # solve the system of equations
  raw, _ = torch.solve(rhs, lhs)
  # write the solution as a single complex matrix
  iC = raw[:C.shape[0], :] + 1j * raw[C.shape[0]:, :]
  return iC

You can verify the solution using numpy:
# C is a complex torch tensor
iC = complex_inverse(C)

with torch.no_grad():
  print(np.isclose(iC.cpu().numpy() @ C.cpu().numpy(), np.eye(C.shape[0])).all())

Note that by using inverse of block-matrices tricks you may reduce the computational cost of the solve operation.


Instruction: Zero diagonal of a PyTorch tensor?

Input:
Is there a simple way to zero the diagonal of a PyTorch tensor?
For example I have:
tensor([[2.7183, 0.4005, 2.7183, 0.5236],
        [0.4005, 2.7183, 0.4004, 1.3469],
        [2.7183, 0.4004, 2.7183, 0.5239],
        [0.5236, 1.3469, 0.5239, 2.7183]])

And I want to get:
tensor([[0.0000, 0.4005, 2.7183, 0.5236],
        [0.4005, 0.0000, 0.4004, 1.3469],
        [2.7183, 0.4004, 0.0000, 0.5239],
        [0.5236, 1.3469, 0.5239, 0.0000]])

Output:
I believe the simplest would be to use torch.diagonal:
z = torch.randn(4,4)
torch.diagonal(z, 0).zero_()
print(z)
>>> tensor([[ 0.0000, -0.6211,  0.1120,  0.8362],
            [-0.1043,  0.0000,  0.1770,  0.4197],
            [ 0.7211,  0.1138,  0.0000, -0.7486],
            [-0.5434, -0.8265, -0.2436,  0.0000]])

This way, the code is perfectly explicit, and you delegate the performance to pytorch's built in functions.


Instruction: RuntimeError: Given groups=1, weight of size [16, 1, 3, 3], expected input[16, 3, 1, 28] to have 1 channels, but got 3 channels instead

Input:
I know my images have only 1 channel so the first conv layer is (1,16,3,1) , but I have no idea why I got such an error.
Here is my code (I post only the related part).
    org_x = train_csv.drop(['id', 'digit', 'letter'], axis=1).values
    org_x = org_x.reshape(-1, 28, 28, 1)
    org_x = org_x/255
    org_x = np.array(org_x)
    org_x = org_x.reshape(-1, 1, 28, 28)
    org_x = torch.Tensor(org_x).float()

    x_test = test_csv.drop(['id','letter'], axis=1).values
    x_test = x_test.reshape(-1, 28, 28, 1)
    x_test = x_test/255
    x_test = np.array(x_test)
    x_test = x_test.reshape(-1, 1, 28, 28)
    x_test = torch.Tensor(x_test).float()

    y = train_csv['digit']
    y = list(y)
    print(len(y))
    org_y = np.zeros([len(y), 1])
    for i in range(len(y)):
        org_y[i] = y[i]
    org_y = np.array(org_y)
    org_y = torch.Tensor(org_y).float()

    from sklearn.model_selection import train_test_split
    x_train, x_valid, y_train, y_valid = train_test_split(
        org_x, org_y, test_size=0.2, random_state=42)

I checked the x_train shape is [1638, 1, 28, 28] and the x_valid shape is [410, 1, 28, 28].
    transform = transforms.Compose([transforms.ToPILImage(),
                            transforms.ToTensor(),
                            transforms.Normalize((0.5, ), (0.5, )) ])

    class kmnistDataset(data.Dataset):
        def __init__(self, images, labels, transforms=None):
            self.x = images
            self.y = labels
            self.transforms = transforms

        def __len__(self):
            return (len(self.x))

        def __getitem__(self, idx):
            data = np.asarray(self.x[idx][0:]).astype(np.uint8)

            if self.transforms:
                data = self.transforms(data)

            if self.y is not None:
                return (data, self.y[idx])
            else:
                return data

    train_data = kmnistDataset(x_train, y_train, transforms=transform)
    valid_data = kmnistDataset(x_valid, y_valid, transforms=transform)

    # dataloaders
    train_loader = DataLoader(train_data, batch_size=16, shuffle=True)
    valid_loader = DataLoader(valid_data, batch_size=16, shuffle = False)

And here is my model
    class Net(nn.Module):
      def __init__(self):
            super(Net, self).__init__()

            self.conv1 = nn.Conv2d(1, 16, 3, padding=1)
            self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
            self.conv3 = nn.Conv2d(32, 64, 3, padding=1)

            self.bn1 = nn.BatchNorm2d(16)
            self.pool = nn.MaxPool2d(2, 2)

            unit = 64 * 14 * 14
            self.fc1 = nn.Linear(unit, 500)
            self.fc2 = nn.Linear(500, 10)

        def forward(self, x):
            x = self.pool(F.relu(self.bn1(self.conv1(x))))
            x = F.relu(self.conv2(x))
            x = F.relu(self.conv3(x))
            x = x.view(-1, 128 * 28 * 28)
            x = F.relu(self.fc1(x))
            x = self.fc2(x)
            return x

    model = Net()
    print(model)

Lastly,
    n_epochs = 30

    valid_loss_min = np.Inf

    for epoch in range(1, n_epochs+1):
        train_loss = 0
        valid_loss = 0

        ###################
        # train the model #
        ###################
        model.train()
        for data in train_loader:
            inputs, labels = data[0], data[1]
            optimizer.zero_grad()
            output = model(inputs)
            loss = criterion(output, labels)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()*data.size(0)

        #####################
        # validate the model#
        #####################
        model.eval()
        for data in valid_loader:
            inputs, labels = data[0], data[1]
            output = model(inputs)
            loss = criterion(output, labels)
            valid_loss += loss.item()*data.size(0)

        train_loss = train_loss/ len(train_loader.dataset)
        valid_loss = valid_loss / len(valid_loader.dataset)

        print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
            epoch, train_loss, valid_loss))

When I run it, I got this error message
RuntimeError: Given groups=1, weight of size [16, 1, 3, 3], expected input[16, 3, 1, 28] to have 1 channels, but got 3 channels instead
To be specific,
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-14-b8783819421f> in <module>
         14         inputs, labels = data[0], data[1]
         15         optimizer.zero_grad()
    ---> 16         output = model(inputs)
         17         loss = criterion(output, labels)
         18         loss.backward()

    /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in   _call_impl(self, *input, **kwargs)
        725             result = self._slow_forward(*input, **kwargs)
        726         else:
    --> 727             result = self.forward(*input, **kwargs)
        728         for hook in itertools.chain(
        729                 _global_forward_hooks.values(),

    <ipython-input-12-500e34c49306> in forward(self, x)
         26
         27     def forward(self, x):
    ---> 28         x = self.pool(F.relu(self.bn1(self.conv1(x))))
         29         x = F.relu(self.conv2(x))
         30         x = F.relu(self.conv3(x))

    /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in         _call_impl(self, *input, **kwargs)
        725             result = self._slow_forward(*input, **kwargs)
        726         else:
    --> 727             result = self.forward(*input, **kwargs)
        728         for hook in itertools.chain(
        729                 _global_forward_hooks.values(),

    /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
        421
        422     def forward(self, input: Tensor) -> Tensor:
    --> 423         return self._conv_forward(input, self.weight)
        424
        425 class Conv3d(_ConvNd):

    /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
        418                             _pair(0), self.dilation, self.groups)
        419         return F.conv2d(input, weight, self.bias, self.stride,
    --> 420                         self.padding, self.dilation, self.groups)
        421
        422     def forward(self, input: Tensor) -> Tensor:

    RuntimeError: Given groups=1, weight of size [16, 1, 3, 3], expected input[16, 3, 1, 28]         to have 1 channels, but got 3 channels instead

Output:
I tried a small demo with your code. and it works fine until your code had x = x.view(-1, 64*14*14) and input shape of torch.Size([1, 1, 28 ,28])
import torch
import torch.nn as nn
import torch.nn.functional as F
class Net(nn.Module):
        def __init__(self):
            super(Net, self).__init__()

            self.conv1 = nn.Conv2d(1, 16, 3, padding=1)
            self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
            self.conv3 = nn.Conv2d(32, 64, 3, padding=1)

            self.bn1 = nn.BatchNorm2d(16)
            self.pool = nn.MaxPool2d(2, 2)

            unit = 64 * 14 * 14
            self.fc1 = nn.Linear(unit, 500)
            self.fc2 = nn.Linear(500, 10)

        def forward(self, x):
            x = self.pool(F.relu(self.bn1(self.conv1(x))))
            x = F.relu(self.conv2(x))
            x = F.relu(self.conv3(x))
            #print(x.shape)
            x = x.view(-1, 64*14*14)
            x = F.relu(self.fc1(x))
            x = self.fc2(x)
            return x

model = Net()
print(model)

data = torch.rand((1,1,28,28))
pred = model(data)

And if i give my data tensor as data = torch.rand((1,3,28,28)) i get your error i.e      RuntimeError: Given groups=1, weight of size [16, 1, 3, 3], expected input[16, 3, 1, 28]         to have 1 channels, but got 3 channels instead
So please check your channel dim of your data just before passing it to your model i.e here  (highlighted by ** **)
for data in train_loader:
        inputs, labels = data[0], data[1]
        optimizer.zero_grad()
        **print(inputs.shape)**
        output = model(inputs)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()*data.size(0)


Instruction: Pytorch CUDA error: no kernel image is available for execution on the device on RTX 3090 with cuda 11.1

Input:
If I run the following:
import torch
import sys
print('A', sys.version)
print('B', torch.__version__)
print('C', torch.cuda.is_available())
print('D', torch.backends.cudnn.enabled)
device = torch.device('cuda')
print('E', torch.cuda.get_device_properties(device))
print('F', torch.tensor([1.0, 2.0]).cuda())

I get this:
A 3.7.5 (default, Nov  7 2019, 10:50:52)
[GCC 8.3.0]
B 1.8.0.dev20210115+cu110
C True
D True
E _CudaDeviceProperties(name='GeForce RTX 3090', major=8, minor=6, total_memory=24267MB, multi_processor_count=82)
F
<stacktrace>
CUDA error: no kernel image is available for execution on the device

More info about my system:

Nvidia version: NVIDIA-SMI 455.38 Driver Version: 455.38 CUDA Version: 11.1
python 3.7, Ubuntu 18.04

Output:
Found a fix for my problem here: https://github.com/pytorch/pytorch/issues/31285#issuecomment-739139454
pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html -U

Then my code snippet gives:
A 3.7.5 (default, Nov  7 2019, 10:50:52)
[GCC 8.3.0]
B 1.8.0.dev20210115+cu110
C True
D True
E _CudaDeviceProperties(name='GeForce RTX 3090', major=8, minor=6, total_memory=24267MB, multi_processor_count=82)
F tensor([1., 2.], device='cuda:0')


Instruction: Reshaping function in numpy

Input:
I am trying to reshape data for image classification purpose. I want to convert shape (32,32,3) to (1,3,32,32). I have used two ways for the reshaping purpose and got different results. The first one is numpy reshape method. The other code is written by me.
def res(t):
    n = np.zeros((3,32,32))

    for j in range(3):
        for k in range(32):
            for l in range(32):
                n[j][k][l]=t[k][l][j]

    n=n.reshape(1,3,32,32)
    return n

I am not able to understand what is the difference between both approaches.

Output:
This is what you want to do with np.reshape after transpose -
new = original.transpose(2,0,1).reshape(1,3,32,32)
#(32,32,3)->(3,32,32)->(1,3,32,32)

##OR##

new = original.transpose(2,0,1)[None,...]
#(32,32,3)->(3,32,32)->(1,3,32,32)

Full code with a comparison of results between your function and the transpose method.
t = np.random.random((32,32,3))

def res(t):
    n = np.zeros((3,32,32))

    for j in range(3):
        for k in range(32):
            for l in range(32):
                n[j,k,l]=t[k,l,j]  #<--- fixed indexing

    n=n.reshape(1,3,32,32)
    return n

## METHOD Transpose and Reshape
np.allclose(t.transpose(2,0,1).reshape(1,3,32,32), res(t))
#True

## METHOD Transpose and new axis
np.allclose(t.transpose(2,0,1)[None,...], res(t))
#True


Instruction: How can I create a tensor of batch batch_size of uniformly distributed values between -1 and 1?

Input:
The title pretty much sums it, I'm trying to implement a GAN:
How can I create a tensor of batch batch_size of uniformly distributed values between -1 and 1 with pytorch?
def create_latent_batch_vectors(batch_size, latent_vector_size, device):
'''
The function creates a random batch of latent vectors with random values
distributed uniformly between -1 and 1.
Finally, it moves the tensor to the given ```device``` (cpu or gpu).
The output should have a shape of [batch_size, latent_vector_size].
'''
# maybe torch.distributions.uniform.Uniform() somehow?
return z.to(device)

Thanks!

Output:
Let us first define an uniform distribution with a low-range as -1 and high-range as +1
dist = torch.distributions.uniform.Uniform(-1,1)
sample_shape = torch.Size([2])
dist.sample(sample_shape)
>tensor([0.7628, 0.3497])

This is a tensor of shape 2 (sample_shape). It doesn't have batch_shape. Let's check:
dist.batch_shape
>torch.Size([])

Now let's use expand. It essentially creates a new distribution instance by expanding the batch_shape.
new_batch_shape = torch.Size([5])   # batch_shape of [5]
expanded_dist = dist.expand(new_batch_shape)

Check:
expanded_dist.batch_shape
>torch.Size([5])

Creating a tensor of shape [batch_size, sample_shape]
expanded_dist.sample(sample_shape)
>tensor([[0.1592, 0.3404, 0.3520, 0.3038, 0.0393],
        [0.9368, 0.0108, 0.5836, 0.6156, 0.6704]])

The three types of shapes are defined as follows:

Sample shape describes independent, identically distributed draws from the distribution.
Batch shape describes independent, not identically distributed draws. Namely, we may have a set of (different)
parameterizations to the same distribution. This enables the common
use case in machine learning of a batch of examples, each modeled by
its own distribution.
Event shape describes the shape of a single draw (event space) from the distribution; it may be dependent across dimensions.


Instruction: Zeroing the diagonal of a matrix by multiplying by (1-I)

Input:
I have a tensor, lets say like this:
tensor([[2.7183, 0.4005, 2.7183, 0.5236],
    [0.4005, 2.7183, 0.4004, 1.3469],
    [2.7183, 0.4004, 2.7183, 0.5239],
    [0.5236, 1.3469, 0.5239, 2.7183]])

And I want to zero its main diagonal by multiplying it by (1-I), meaning by 1 minus the
identity matrix.
How can I do this in pytorch?
Result of the example should be:
    tensor([[0.0000, 0.4005, 2.7183, 0.5236],
    [0.4005, 0.0000, 0.4004, 1.3469],
    [2.7183, 0.4004, 0.0000, 0.5239],
    [0.5236, 1.3469, 0.5239, 0.0000]])

I'm looking for a general case solution and not specific to the example I gave.
Thanks!

Output:
torch.eye will be helpful for generating identity matrix
import torch

x = torch.tensor([[2.7183, 0.4005, 2.7183, 0.5236],
    [0.4005, 2.7183, 0.4004, 1.3469],
    [2.7183, 0.4004, 2.7183, 0.5239],
    [0.5236, 1.3469, 0.5239, 2.7183]],dtype=torch.float32)
y = 1-torch.eye(x.size()[0],dtype=torch.float32)  #only if x is square matrix
output = x*y


Instruction: ModuleNotFoundError: No module named 'numpy.core._multiarray_umath' on matplotlib import

Input:
I'm trying to run a simple testfile on a remote Server. But it throws a numpy error for matplotlib.pyplot. Here is the code
import matplotlib.pyplot as plt
import numpy as np

# Fixing random state for reproducibility
np.random.seed(19680801)

x, y = np.random.randn(2, 100)
print('x')
print(x)
print('y')
print(y)
fig, [ax1, ax2] = plt.subplots(2, 1, sharex=True)
ax1.xcorr(x, y, usevlines=True, maxlags=50, normed=True, lw=2)
ax1.grid(True)

ax2.acorr(x, usevlines=True, normed=True, maxlags=50, lw=2)
ax2.grid(True)

plt.show()

Here is the error message.

PyTorch/1.7-py36-cuda11/numpy/core/overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "graph_test.py", line 1, in
import matplotlib.pyplot as plt

/PyTorch/1.7-py36-cuda11/numpy/core/init.py", line 48, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:

The Python version is: Python3.7 from "/projects/smiles/Model/venv/bin/python"
The NumPy version is: "1.19.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy.core._multiarray_umath'

Python Version: 3.7.5
Numpy Version: 1.19.4
Matplotlib version: 3.3.3

Output:
According to the site given by the library numpy:
to fix the error you need to check couple things that commonly gives this error:
1- Check if you are using the right version of python and the right version of numpy, check the documentation fo further information.(you might also have multiple versions of python that can affect the run of your libraries)+ try the newer version of numpy 1.19.5
2- check if your python path is in the environment variables of your system.
you can check your path by running this command in your terminal:
import os

print("PYTHONPATH:", os.environ.get('PYTHONPATH'))

print("PATH:", os.environ.get('PATH'))

3- try to uninstall and reinstall the numpy library(make sure to install the right version to satisfy what ever program you want to run)
4- if any of the above solutions didn't work, I highly recomend to install the anaconda installer here is a link to it : https://www.anaconda.com/products/individual
and that because anaconda most likely solve the problems of downloading libraries/ errors you get from those libraries.
Hopefully those solutions helped you, and feel free to ask any question regarding it.


Instruction: PyTorch out of GPU memory in test loop

Input:
For the following training program, training and validation are all ok.
Once reach to Test method, I have CUDA out of memory. What should I change so that I have enough memory to test as well.
import torch
from torchvision import datasets, transforms
import torch.nn.functional as f
class CnnLstm(nn.Module):
    def __init__(self):
        super(CnnLstm, self).__init__()
        self.cnn = CNN()
        self.rnn = nn.LSTM(input_size=180000, hidden_size=256, num_layers=2, batch_first=True)#stacked LSTM with 2 layers
        #print(num_classes)
        self.linear = nn.Linear(256, num_classes)
        #print('after num_classes')

    def forward(self, x):
        #print(x.shape)
        batch_size, time_steps, channels, height, width = x.size()
        c_in = x.view(batch_size * time_steps, channels, height, width)
        _, c_out = self.cnn(c_in)
        r_in = c_out.view(batch_size, time_steps, -1)
        r_out, (_, _) = self.rnn(r_in)
        r_out2 = self.linear(r_out[:, -1, :])
        return f.log_softmax(r_out2, dim=1)

class TrainCNNLSTM:
    def __init__(self):
        self.seed = 1
        self.batch_size = 8
        self.validate_batch_size = 8
        self.test_batch_size = 1
        self.epoch = 20
        self.learning_rate = 0.01
        self.step = 100
        self.train_loader = None
        self.validate_loader = None
        self.test_loader = None
        #print('before')
        self.model = CnnLstm().to(device)
        #print('after')
        self.criterion = nn.CrossEntropyLoss()

    def load_data(self):
        data_loader = DataLoader()
        self.train_loader = data_loader.get_train_data(self.batch_size)
        self.validate_loader = data_loader.get_validate_data(self.validate_batch_size)
        self.test_loader = data_loader.get_test_data(self.test_batch_size)

    def train(self):
        optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate, momentum=0.9)
        scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=self.learning_rate/100.0, max_lr=self.learning_rate, step_size_up=13)
        #optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate)
        for epoch in range(self.epoch):
            t_losses=[]
            for iteration, (data, target) in enumerate(self.train_loader):
                data = np.expand_dims(data, axis=1)
                data = torch.FloatTensor(data)
                data, target = data.cuda(), target.cuda()
                data, target = Variable(data), Variable(target)
                optimizer.zero_grad()
                output = self.model(data)
                loss = self.criterion(output, target)
                #loss = f.nll_loss(output, target)
                t_losses.append(loss)
                loss.backward()
                optimizer.step()
                scheduler.step()
                if iteration % self.step == 0:
                   print('Epoch: {} | train loss: {:.4f}'.format(epoch, loss.item()))
            avgd_trainloss = sum(t_losses)/len(t_losses)
            self.validate(epoch, avgd_trainloss)

    def validate(self, epoch, avg_tloss):
        v_losses=[]
        with torch.no_grad():
            for iteration, (data, target) in enumerate(self.validate_loader):
                data = np.expand_dims(data, axis=1)
                data = torch.FloatTensor(data)
                data, target = data.cuda(), target.cuda()
                data, target = Variable(data), Variable(target)
                output = self.model(data)
                loss = self.criterion(output, target)
                #loss = f.nll_loss(output, target)
                v_losses.append(loss)
        avgd_validloss = sum(v_losses)/len(v_losses)
        print('Epoch: {} | train loss: {:.4f} | validate loss: {:.4f}'.format(epoch, avg_tloss, avgd_validloss))

    def test(self):
        test_loss = []
        correct = 0
        for data, target in self.test_loader:
            data = np.expand_dims(data, axis=1)
            data = torch.FloatTensor(data)
            data, target = data.cuda(), target.cuda()
            data, target = Variable(data, volatile=True), Variable(target)
            output = self.model(data)
            loss = self.criterion(output, target)
            #f.nll_loss(output, target, size_average=False).item()  # sum up batch loss
            test_loss.append(loss)
            pred = torch.max(output, 1)[1].data.squeeze()
            correct += pred.eq(target.data.view_as(pred)).long().cpu().sum()

        test_loss = sum(test_loss)/len(test_loss)
        print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(test_loss, correct, len(self.test_loader.dataset),
                100. * correct / len(self.test_loader.dataset)))

train = TrainCNNLSTM()
train.load_data()
train.train()
train.test()

Output:
You should call .item() on your loss when appending it to the list of losses:
loss = self.criterion(output, target)
test_loss.append(loss.item())

This avoids accumulating tensors in a list which are still attached to the computational graph. I would say the same for your accuracy.


Instruction: PyTorch GPU memory management

Input:
In my code, I want to replace values in the tensor given values of some indices are zero, for example
target_mac_out[avail_actions[:, 1:] == 0] = -9999999

But, it returns OOM
RuntimeError: CUDA out of memory. Tried to allocate 166.00 MiB (GPU 0; 10.76 GiB total capacity; 9.45 GiB already allocated; 4.75 MiB free; 9.71 GiB reserved in total by PyTorch)

I think there is no memory allocation because it just visits the tensor of target_mac_out and check the value and replace a new value for some indices.
Am I understanding right?

Output: It's hard to guess since we do not even know the sizes if the involved tensors, but your indexing avail_actions[:, 1:] == 0 creates a temporary tensor that does require memory allocation.


Instruction: Learning rate finder for CNNLstm model

Input:
I have CNNLstm model as follows.
   class CNN(nn.Module):
        def __init__(self):
            super(CNN, self).__init__()
            self.conv1 = nn.Sequential(
                nn.Conv2d(
                    in_channels=3,
                    out_channels=16,
                    kernel_size=5,
                    stride=1,
                    padding=2,
                ),
                nn.ReLU(),
                nn.MaxPool2d(kernel_size=2),
            )
            self.conv2 = nn.Sequential(
                nn.Conv2d(16, 32, 5, 1, 2),
                nn.ReLU(),
                nn.MaxPool2d(kernel_size=2),
            )
            #print(num_classes)
            self.out = nn.Linear(32 * 75 * 75, num_classes)#32 * 75 * 75/64 * 37 * 37/128 * 18 * 18

        def forward(self, x):
            x = self.conv1(x)
            x = self.conv2(x)
            x = x.view(x.size(0), -1)
            output = self.out(x)
            return output, x

    import torch
    from torchvision import datasets, transforms
    import torch.nn.functional as f
    from torch_lr_finder import LRFinder
    class CnnLstm(nn.Module):
        def __init__(self):
            super(CnnLstm, self).__init__()
            self.cnn = CNN()
            self.rnn = nn.LSTM(input_size=180000, hidden_size=256, num_layers=3, batch_first=True)#stacked LSTM with 2 layers
            self.linear = nn.Linear(256, num_classes)

        def forward(self, x):
            batch_size, time_steps, channels, height, width = x.size()
            c_in = x.view(batch_size * time_steps, channels, height, width)
            _, c_out = self.cnn(c_in)
            r_in = c_out.view(batch_size, time_steps, -1)
            r_out, (_, _) = self.rnn(r_in)
            r_out2 = self.linear(r_out[:, -1, :])
            return f.log_softmax(r_out2, dim=1)

class TrainCNNLSTM:
    def __init__(self):
        self.seed = 1
        self.batch_size = 8
        self.validate_batch_size = 8
        self.test_batch_size = 1
        self.epoch = 50
        self.learning_rate = 0.005
        self.step = 100
        self.train_loader = None
        self.validate_loader = None
        self.test_loader = None
        self.modelloaded = False
        self.model = CnnLstm().to(device)
        self.criterion = nn.CrossEntropyLoss()
        #self.optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate)#self.learning_rate = 0.001
        self.optimizer = torch.optim.AdamW(self.model.parameters())
        #self.scheduler = optim.lr_scheduler.OneCycleLR(self.optimizer, 2e-3, epochs=self.epoch , steps_per_epoch=len(train_loader))

    def load_data(self):
        data_loader = DataLoader()
        self.train_loader = data_loader.get_train_data(self.batch_size)
        self.validate_loader = data_loader.get_validate_data(self.validate_batch_size)
        self.test_loader = data_loader.get_test_data(self.test_batch_size)

    def do_lrfinder(self):
        lr_finder = LRFinder(self.model, self.optimizer, self.criterion, device)
        lr_finder.range_test(self.train_loader, end_lr=1, num_iter=1000)
        lr_finder.plot()
        plt.savefig("LRvsLoss.png")
        plt.close()

    def train(self):
        for epoch in range(0, self.epoch):
            t_losses=[]
            for iteration, (data, target) in enumerate(self.train_loader):
                print(data.shape)
                data = np.expand_dims(data, axis=1)
                print(data.shape)
                data = torch.FloatTensor(data)
                data, target = data.cuda(), target.cuda()
                data, target = Variable(data), Variable(target)
                self.optimizer.zero_grad()

Since it is CNNLstm model, the data input shape to the model are batch_size, time_steps, channels, height, width.
(8, 1, 3, 300, 300)
To use torch_lr_finder, we need to run the following code.
lr_finder = LRFinder(self.model, self.optimizer, self.criterion, device)
lr_finder.range_test(self.train_loader, end_lr=1, num_iter=1000)

self.train_loader output shape is (8, 3, 300, 300). So during finding learning rate, self.model can't be used.
How can I use torch_lr_finder for such model?

Output:
One possibility is that instead of expanding the dims in the for loop you could pass the tensor into the forward function of the model and just use .unsqueeze(1) there. Like this
                print(data.shape)
                print(data.shape)
                data = torch.FloatTensor(data)

just omit the expand dims then in your forward function do this
x = x.unsqueeze(1)


Instruction: Pandas Dataframe to tensor

Input:
I have a dataframe with 3 columns (a date index, a price and a string symbol).
It looks like that:

Date
Price
Symbol

2019-01-02
39.480000
AAPL

2019-01-02
101.120003
MSFT

2019-01-02
62.023998
TSLA

2019-01-03
35.547501
AAPL

2019-01-03
97.400002
MSFT

2019-01-03
60.071999
TSLA

I'm looking for some panda/pytorch/python syntactic sugar to turn that into a tensor/matrix that will be:
[ [ 39.480000, 101.120003, 62.023998], [35.547501, 97.400002, 60.071999]]

With the number length of the first dimension will be the number of unique dates, and the length of the second will be the number of unique symbols.
I'm guaranteed to have exactly 3 symbols per date and I want that each row of my matrix follow the same order for its columns (e.g always AAPL, MSFT, TSLA).
Now, that is very easy with some for loops, but I'm looking for something more "pythonic"

Output:
You can groupby the date column, convert the groups of Price to numpy arrays, and then convert this series to a tensor:
import torch
import pandas as pd

prices = df.groupby(['Date'])['Price'].apply(np.array)
my_tensor = torch.tensor(prices)


Instruction: 2D times 2D equals a 3d pytorch tensor

Input:
Given two 2-D pytorch tensors:
A = torch.FloatTensor([[1,2],[3,4]])
B = torch.FloatTensor([[0,0],[1,1],[2,2]])

Is there an efficient way to calculate a tensor of shape (6, 2, 2) where each entry is a column of A times each row of B?
For example, with A and B above, the 3D tensor should have the following matrices:
[[[0, 0],
  [0, 0]],

 [[1, 1],
  [3, 3]],

 [[2, 2],
  [6, 6]],

 [[0, 0],
  [0, 0]],

 [[2, 2],
  [4, 4]],

 [[4, 4],
  [8, 8]]]

I know how to do it via for-loop but I am wondering if could have an efficient way to save it.

Output:
Pytorch tensors implement numpy style broadcast semantics which will work for this problem.
It's not clear from the question if you want to perform matrix multiplication or element-wise multiplication. In the length 2 case that you showed the two are equivalent, but this is certainly not true for higher dimensionality! Thankfully the code is almost the same so I'll just give both options.
A = torch.FloatTensor([[1, 2], [3, 4]])
B = torch.FloatTensor([[0, 0], [1, 1], [2, 2]])

# matrix multiplication
C_mm = (A.T[:, None, :, None] @ B[None, :, None, :]).flatten(0, 1)

# element-wise multiplication
C_ew = (A.T[:, None, :, None] * B[None, :, None, :]).flatten(0, 1)

Code description. A.T transposes A and the indexing with None inserts unitary dimensions so A.T[:, None, :, None] will be shape (2, 1, 2, 1) and B[None, :, None, :] is shape (1, 3, 1, 2). Since @ (matrix multiplication) operates on the last two dimensions of tensors, and broadcasts the other dimensions, then the result is matrix multiplication for each column of A times each row of B. In the element-wise case the broadcasting is performed on every dimension. The result is a (2, 3, 2, 2) tensor. To turn it into a (6, 2, 2) tensor we just flatten the first two dimensions using Tensor.flatten.


Instruction: Where to implement pre-processing in PyTorch Lightning (e.g. tokenizing input text)

Input:
Is there a convention to implement some kind of predict() method in PyTorch Lightning that does pre-processing before performing the actual prediction using forward()?
In my case, I have a text classifier consisting of an embedding layer and a few fully connected layers. The text needs to be tokenized before being passed to the embedding layer. During training and evaluation the LightningDataModule's setup() method does the job.
Now, I'm wondering what the best practice for inference during production is. I could add a predict() method to my LightningModule where I could write the same pre-processing code as in LightningDataModule.setup(). But, of course, I do not want to duplicate the code.
In this community example project linked in the official PyTorch Lightning docs, the authors define a prepare_sample() function in the LightningModule that is used by their predict() function, and is also passed to the LightningDataModule.
Is this the right way to handle pre-processing? Also, why is there no prepare_sample() or predict() in LightningModule? To me, this seems like a common use case, for example:
model = load_model('data/model.ckpt')  # load pre-trained model, analyzes user reviews

user_input = input('Your movie review > ')

predicted_rating = model.predict(user_input)  # e.g. "I liked the movie pretty much." -> 4 stars

print('Predicted rating: %s/5 stars' % predicted_rating)

Now that I think about it, predict() should also process the result from forward() the same way the evaluation code does, like selecting the class with the highest output or selecting all classes with outputs larger than some threshold - some more code that should not be duplicated.

Output:
Why do you use a LightningModule if the code should be for production? If the model is finished you only need to load the model from memory and define the preprocess steps.
The repository you refer to have implemented the predict, and prepare_sample on top of the LightningModule.
In my opinion pytorch-lightning is for training and evaluation of the model and not for production. We would not want to keep the analytics and debugging when sending a model to production so instead we create a slimmed version which only have loading of model, preprocess and prediction.
Towardsdatascience have a small code example: https://towardsdatascience.com/how-to-deploy-pytorch-lightning-models-to-production-7e887d69109f


Instruction: tensorflow autodiff slower than pytorch's counterpart

Input:
I am using tensorflow 2.0 and trying to evaluate gradients for backpropagating to a simple feedforward neural network. Here's how my model looks like:
def __init__(self, input_size, output_size):
        inputs = tf.keras.Input(shape=(input_size,))
        hidden_layer1 = tf.keras.layers.Dense(30, activation='relu')(inputs)
        outputs = tf.keras.layers.Dense(output_size)(hidden_layer1)
        self.model = tf.keras.Model(inputs=inputs, outputs=outputs)

        self.optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
        self.loss_function = tf.keras.losses.Huber()

The forward pass to this network is fine but when I use gradient tape to train the model, it is at least 10x slower than PyTorch.
Training function:
def learn_modified_x(self, inputs, targets, actions):
        with tf.GradientTape() as tape:
            predictions = self.model(inputs)
            predictions_for_action = gather_single_along_axis(predictions, actions)
            loss = self.loss_function(targets, predictions_for_action)

        grads = tape.gradient(loss, self.model.trainable_weights)
        self.optimizer.apply_gradients(zip(grads, self.model.trainable_weights))

I tried commenting lines to find what is actually causing the problem. I discovered that tape.gradient is a significant contributor to this situation.
Any idea?
PyTorch implementation
    def __init__(self, input_size, nb_action):
        super(Network, self).__init__()
        self.input_size = input_size
        self.nb_action = nb_action
        self.fc1 = nn.Linear(input_size, 30)
        self.fc2 = nn.Linear(30, nb_action)

    def forward(self, state):
        x = F.relu(self.fc1(state))
        q_values = self.fc2(x)
        return q_values

    def learn(self, batch_state, batch_next_state, batch_reward, batch_action):
        outputs = self.model(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)
        next_outputs = self.model(batch_next_state).detach().max(1)[0]
        target = self.gamma*next_outputs + batch_reward
        td_loss = F.smooth_l1_loss(outputs, target)
        self.optimizer.zero_grad()
        td_loss.backward(retain_variables = True)
        self.optimizer.step()

Output:
def __init__(self,...):
  ...
  self.model.call = tf.function(self.model.call)
  ...

you need use tf.function to wrap your model's call function.


Instruction: Open3D-ML and pytorch

Input:
I’m currently trying to work with open3d ML and Pytorch. I followed the installation guide given in the Open3D-ML github. However when I try to import open3d.ml.torch it sends me the following error : Exception: Open3D was not built with PyTorch support!
I’m working with
python 3.8
open3d 0.12.0
pytorch 1.6.0
cuda 10.1
Windows 10

Do you have any idea of where that error comes from ?

Output: It does not support for Windows at the moment. You can install Ubuntu on WSL (Window Subsystem for Linux) on Windows OS, and install open3d-ml on ubuntu.


Instruction: Pytorch model 2D regression given an scalar input

Input:
I want to create a model to perform this regression:

My dataset looks like:
t,x,y
0.0,-,0.5759052335487023
0.01,-,-
0.02,1.1159124144549086,-
0.03,-,-
0.04,1.0054825084650338,0.4775267298487888
0.05,-,-

I'm having some troubles with loss, dataset load, batch_size, and Net structure (I add one single layer to simplify the problem)
Thats my code:
Net:
class Net(nn.Module):
    '''Model to regress 2d time series values given scalar input.'''
    def __init__(self):
        super(Net, self).__init__()
        #Layers
        self.predict = nn.Linear(1, 2)

    def forward(self, x):
        x = self.predict(x)
        return x

Dataset load
class TimeSeriesDataset(torch.utils.data.Dataset):

    def __init__(self, csv_file):
        #Load the dataset
        #Load the csv file as a dataframe
        df = pd.read_csv(csv_file, header=0, na_values='-')
        #Store the inputs and outputs
        self.x = df.values[:,:-2].astype('float32')
        self.y = df.values[:,1:].astype('float32')
        #Ensure target has the right shape
        self.y = self.y.reshape((len(self.y),2))

    def __len__(self):
        #Return the number of rows in the dataset
        return len(self.x)

    def __getitem__(self, idx):
        #Return a row at an index
        return [self.x[idx], self.y[idx]]

Trainloader, loss, optimizer
dataset = TimeSeriesDataset('data.csv')
trainloader = torch.utils.data.DataLoader(
    dataset, batch_size=32, shuffle=True, num_workers=2)

def lossFunc(outputs, labels):
  # nn.MSELoss() #Mean Squared Error, works fine with regression problems and with small numbers (x-y)^2
  return torch.mean((outputs-labels)**2)

net = Net()
optimizer = torch.optim.SGD(net.parameters(), lr=0.01)
print(net)

Trainning:
for epoch in range(300):

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # TODO get the data
        # inputs, labels
        inputs, labels = data
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        #print("Inputs", inputs)
        #print("labels", labels)
        #print("outputs", outputs)
        loss = lossFunc(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 20 == 19:    # print every 20 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 20))
            running_loss = 0.0

print('Finished Training')

Outputs looks this way:
tensor([[nan, nan],
        [nan, nan],
        [nan, nan],
...

And when I execute the 300 epochs error value doesn't change and prints nan

Output:
After the line loss = loss(outputs, labels), loss is now a tensor, not a function anymore. Python does not allow you to have distinct objects with identical names.
So after the first call, loss has become a tensor, and as the error says "tensors are not callable", so the second call fails


Instruction: How to initialize columns in hybrid sparse tensor

Input:
How initialize in pytorch hybrid tensor torch.sparse_coo_tensor (one dimension is sparse and other is not), which have the following dense representation?
array([[1, 0, 5, 0],
       [2, 0, 6, 0],
       [3, 0, 7, 0],
       [4, 0, 8, 0]])

What should I put into the indices argument?

Output:
How to initialize
Something like this:
import torch

indices = torch.tensor([[0, 0, 1, 1, 2, 2, 3, 3], [0, 2, 0, 2, 0, 2, 0, 2]])
tensor = torch.sparse_coo_tensor(
    indices, torch.tensor([1, 2, 3, 4, 5, 6, 7, 8]), size=(4, 4)
)

Given above:

indices - first dimension specifies row, second column, where non-zero value(s) will be located. Those become pairs, in this case: (0, 0), (0, 2), (1, 0), (1, 2)... and so on
values - values located at those pairs, so 1 will be under (0, 0) coordinate, 2 under (0, 2) and so it goes.
size - total size of the matrix, optional, might be inferred in this case from your input

8 pairs, 8 values, there are also other ways to specify it, but the idea holds.
And a quick check:
print(tensor)

print(tensor.to_dense())

Gives us:
tensor(indices=tensor([[0, 0, 1, 1, 2, 2, 3, 3],
                       [0, 2, 0, 2, 0, 2, 0, 2]]),
       values=tensor([1, 2, 3, 4, 5, 6, 7, 8]),
       size=(4, 4), nnz=8, layout=torch.sparse_coo)
tensor([[1, 0, 2, 0],
        [3, 0, 4, 0],
        [5, 0, 6, 0],
        [7, 0, 8, 0]])

Why to initialize
If your actual data is 50% sparse, you shouldn't use COO tensor.
It will save some memory, but operations will be way slower, so keep that in mind.


Instruction: How to select indices according to another tensor in pytorch

Input:
The task seems to be simple, but I cannot figure out how to do it.
So what I have are two tensors:

an indices tensor indices with shape (2, 5, 2), where the last dimensions corresponds to indices in x and y dimension
a "value tensor" value with shape (2, 5, 2, 16, 16), where I want the last two dimensions to be selected with x and y indices

To be more concrete, the indices are between 0 and 15 and I want to get an output:
out = value[:, :, :, x_indices, y_indices]

The shape of the output should therefore be of (2, 5, 2). Can anybody help me here? Thanks a lot!
Edit:
I tried the suggestion with gather, but unfortunately it does not seem to work (I changed the dimensions, but it doesn't matter):
First I generate a coordinate grid:
y_t = torch.linspace(-1., 1., 16, device='cpu').reshape(16, 1).repeat(1, 16).unsqueeze(-1)
x_t = torch.linspace(-1., 1., 16, device='cpu').reshape(1, 16).repeat(16, 1).unsqueeze(-1)
grid = torch.cat((y_t, x_t), dim=-1).permute(2, 0, 1).unsqueeze(0)
grid = grid.unsqueeze(1).repeat(1, 3, 1, 1, 1)

In the next step, I am creating some indices. In this case, I always take index 1:
indices = torch.ones([1, 3, 2], dtype=torch.int64)

Next, I am using your method:
indices = indices.unsqueeze(-1).unsqueeze(-1)
new_coords = torch.gather(grid, -1, indices).squeeze(-1).squeeze(-1)

Finally, I manually select index 1 for x and y coordinate:
new_coords_manual = grid[:, :, :, 1, 1]

This outputs the following new coordinates:
new_coords
tensor([[[-1.0000, -0.8667],
         [-1.0000, -0.8667],
         [-1.0000, -0.8667]]])

new_coords_manual
tensor([[[-0.8667, -0.8667],
         [-0.8667, -0.8667],
         [-0.8667, -0.8667]]])

As you can see, it only works for one dimension. Do you have an idea how to fix that?

Output:
What you could do is flatten the first three axes together and apply torch.gather:
>>> grid.flatten(start_dim=0, end_dim=2).shape
torch.Size([6, 16, 16])

>>> torch.gather(grid.flatten(0, 2), axis=1, indices)
tensor([[[-0.8667, -0.8667],
         [-0.8667, -0.8667],
         [-0.8667, -0.8667]]])

As explained on the documentation page, this will perform:

out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1


Instruction: RuntimeError: Input tensor at index 3 has invalid shape [2, 2, 16, 128, 64] but expected [2, 4, 16, 128, 64]

Input:
Runtime error while finetuning a pretrained GPT2-medium model using Huggingface library in SageMaker - ml.p3.8xlarge instance.
The finetuning_gpt2_script.py contains the below,
Libraries:
from transformers import Trainer, TrainingArguments
from transformers import EarlyStoppingCallback
from transformers import GPT2LMHeadModel, GPT2Tokenizer
from transformers import TextDataset,DataCollatorForLanguageModeling

Pretrained Models:
gpt2_model = GPT2LMHeadModel.from_pretrained("gpt2-medium")
gpt2_tokenizer = GPT2Tokenizer.from_pretrained("gpt2-medium")

Train and Test Data Construction:
train_dataset = TextDataset(
          tokenizer=gpt2_tokenizer,
          file_path=train_path,
          block_size=128)

test_dataset = TextDataset(
          tokenizer=gpt2_tokenizer,
          file_path=test_path,
          block_size=128)

data_collator = DataCollatorForLanguageModeling(
        tokenizer=gpt2_tokenizer, mlm=False,
    )

train_path & test_path are unstructured text data file of size 1.45 Million and 200K lines of data
Training arguments:
training_args = TrainingArguments(
        output_dir="./gpt2-finetuned-models", #The output directory
        overwrite_output_dir=True, #overwrite the content of the output directory
        num_train_epochs=1, # number of training epochs
        per_device_train_batch_size=8, # batch size for training #32
        per_device_eval_batch_size=8,  # batch size for evaluation #64
        save_steps=100, # after # steps model is saved
        warmup_steps=500,# number of warmup steps for learning rate scheduler
        prediction_loss_only=True,
        metric_for_best_model = "eval_loss",
        load_best_model_at_end = True,
        evaluation_strategy="epoch",
    )

training_args are the training arguments constructed to train the model.
Trainer:
trainer = Trainer(
        model=gpt2_model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
        callbacks = [early_stop_callback],
    )
early_stop_callback = EarlyStoppingCallback(early_stopping_patience  = 3)

Training:
trainer.train()
trainer.save_model(model_path)

Here, the training is done for only 1 epoch in 4 GPUS using ml.p3.8xlarge instance.
The training is done by torch-distribution like below,
python -m torch.distributed.launch finetuning_gpt2_script.py

While training at the end of the epoch, observed the below error,
RuntimeError: Input tensor at index 3 has invalid shape [2, 2, 16, 128, 64] but expected [2, 4, 16, 128, 64]

Is the RuntimeError because of the way the train_dataset and test_datasetconstructed using TextData ?
Am I doing wrong in the torch-distribution ?

Output:
It could be related to a mismatch in the batch size (expecting a batch size of 4 but receiving a batch size of 2) as suggested here ? Solution provided is to set the parameter drop_last in your DataLoader like this:
tain_text = DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True, drop_last=True)


Instruction: Pytorch Global Pruning is not reducing the size of the model

Input:
I am trying to Prune my Deep Learning model via Global Pruning. The original UnPruned model is about 77.5 MB. However after pruning, when I am saving the model, the size of the model is the same as the original. Can anyone help me with this issue?
Below is the Pruning code:-
import torch.nn.utils.prune as prune

parameters_to_prune = (
(model.encoder[0], ‘weight’),
(model.up_conv1[0], ‘weight’),
(model.up_conv2[0], ‘weight’),
(model.up_conv3[0], ‘weight’),
)
print(parameters_to_prune)

prune.global_unstructured(
parameters_to_prune,
pruning_method=prune.L1Unstructured,
amount=0.2,
)

print(
“Sparsity in Encoder.weight: {:.2f}%”.format(
100. * float(torch.sum(model.encoder[0].weight == 0))
/ float(model.encoder[0].weight.nelement())
)
)
print(
“Sparsity in up_conv1.weight: {:.2f}%”.format(
100. * float(torch.sum(model.up_conv1[0].weight == 0))
/ float(model.up_conv1[0].weight.nelement())
)
)
print(
“Sparsity in up_conv2.weight: {:.2f}%”.format(
100. * float(torch.sum(model.up_conv2[0].weight == 0))
/ float(model.up_conv2[0].weight.nelement())
)
)
print(
“Sparsity in up_conv3.weight: {:.2f}%”.format(
100. * float(torch.sum(model.up_conv3[0].weight == 0))
/ float(model.up_conv3[0].weight.nelement())
)
)

print(
“Global sparsity: {:.2f}%”.format(
100. * float(
torch.sum(model.encoder[0].weight == 0)
+ torch.sum(model.up_conv1[0].weight == 0)
+ torch.sum(model.up_conv2[0].weight == 0)
+ torch.sum(model.up_conv3[0].weight == 0)
)
/ float(
model.encoder[0].weight.nelement()
+ model.up_conv1[0].weight.nelement()
+ model.up_conv2[0].weight.nelement()
+ model.up_conv3[0].weight.nelement()
)
)
)

**Setting Pruning to Permanent**
prune.remove(model.encoder[0], “weight”)
prune.remove(model.up_conv1[0], “weight”)
prune.remove(model.up_conv2[0], “weight”)
prune.remove(model.up_conv3[0], “weight”)

**Saving the model**
PATH = “C:\PrunedNet.pt”
torch.save(model.state_dict(), PATH)

Output:
Prunning won't change the model size if applied like this.
If you have a tensor, say something like:
[1., 2., 3., 4., 5., 6., 7., 8.]

And you prune 50% of data, so for example this:
[1., 2., 0., 4., 0., 6., 0., 0.]

You will still have 8 float values and their size will be the same.
When prunning reduces model size?

When we save weights in a sparse format, but it should have high sparsity (so 10% non-zero elements)
When we actually remove something (like a kernel from Conv2d, it could be removed if it's weights are zero or negligible)

Otherwise it's not going to work. Check out some related projects that would allow you to do it without coding it in on your own, for example Torch-Pruning.


Instruction: Pytorch/cuda : CPU error and map_location

Input:
I write this code to download my model :
args = parser.parse_args()

use_cuda = torch.cuda.is_available()

state_dict = torch.load(args.model)
model = Net()
model.load_state_dict(state_dict)
model.eval()

if use_cuda:
    print('Using GPU')
    model.cuda()
else:
    print('Using CPU')

But my terminal returns the following error   RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
So then I tried to write without really understanding too much :
args = parser.parse_args()

map_location=torch.device('cpu')
state_dict = torch.load(args.model)
model = Net()
model.load_state_dict(state_dict)
model.eval()

But I still have the same mistake. Do you see please how I can correct it? (actually I want to load my model with my CPU).

Output:
I'm assuming you saved the model on a computer with a GPU and are now loading it on a computer without one, or maybe you for some reason the GPU is not available. Also, which line is causing the error?
The parameter map_location needs to be set inside torch.load. Like this:
state_dict = torch.load(args.model, map_location='cpu')

or
map_location=torch.device('cpu')
state_dict = torch.load(args.model, map_location=map_location)

Notice that you need to send the map_location variable to the torch.load function.


Instruction: Implementations and strategies for fast 2D interpolation from irregularly spaced points

Input:
Given a large (~10 million) number of irregularly spaced points in two dimensions, where each point has some intensity ("weight") associated with it, what existing python implementations are there for interpolating the value at:

a specific point at some random position (i.e. point = (0.5, 0.8))
a large number of points at random positions (i.e. points = np.random.random((1_000_000, 2)))
a regular grid at integer positions (i.e. np.indices((1000, 1000)).T)

I am aware that Delaunay triangulation is often used for this purpose. Are there alternatives to doing it this way?
Do any solutions take advantage of multiple CPU cores or GPUs?
As an example, here is an approach using scipy's LinearNDInterpolator. It does not appear to use more than one CPU core.
There are also other options in scipy, but with this question I am especially interested in hearing about other solutions than the ones in scipy.
# The %time tags are IPython magic functions that time that specific line
dimension_shape = (1000, 1000) # we spread the random [0-1] over [0-1000] to avoid floating point errors
N_points = dimension_shape[0] * dimension_shape[1]

known_points = np.random.random((N_points, 2)) * dimension_shape
known_weights = np.random.random((N_points,))

unknown_point = (0.5, 0.8)
unknown_points = np.random.random((N_points, 2)) * dimension_shape
unknown_grid = np.indices(dimension_shape, dtype=float).T.reshape((-1, 2)) # reshape to a list of 2D points

%time tesselation = Delaunay(known_points) # create grid to know neighbours # 6 sec
%time interp_func = LinearNDInterpolator(tesselation, known_weights) # 1 ms
%time interp_func(unknown_point) # 2 sec # run it once because the scipy function needs to compile

%time interp_func(unknown_point) # ~ns
%time interp_func(unknown_grid) # 400 ms
%time interp_func(unknown_points) # 1 min 13 sec

# Below I sort the above `unknown_points` array, and try again
%time ind = np.lexsort(np.transpose(unknown_points)[::-1]) # 306 ms
unknown_points_sorted = unknown_points[ind].copy()

%time interp_func(unknown_points_sorted) # 19 sec <- much less than 1 min!

In the above code, things that take an appreciable amount of time are the construction of the Delaunay grid, and interpolation on a non-regular grid of points. Note that sorting the non-regular points first results in a significant speed improvement!
Do not feel the need to give a complete answer from the start. Tackling any aspect of the above is welcome.

Output:
Scipy is pretty good and I don't think that there are better solutions in Python, but I can add a couple things that might be helpful to you.  First off, your idea of sorting the points is a really good one. The so-called "incremental algorithms" build the Delaunay by inserting vertices one at a time. The first step in inserting a vertex in an existing mesh is to figure out which triangle in the mesh to insert it into. To speed things up, some algorithms start the search right at the point where the most recent insertion occurred.  So if your points are ordered so that each point inserted is relatively close to the previous one, the search is much faster.  If you want more details, you can look up the "Lawson's Walk" algorithm.   In my own implementation of the Delaunay (which is in Java, so I'm afraid it won't help you), I have a sort based on the Hilbert space-filling curve. the Hilbert sort works great. But even just sorting by x/y coordinates is a help.
In terms of whether there are other ways to interpolate without using the Delaunay... You could try something using Inverse-Distance-Weighting (IDW).  IDW techniques don't require the Delaunay, but they do require some way to figure out which vertices are close to the point for which you wish to interpolate. I've played with dividing my coordinate space into uniformly spaced bins, storing the vertices in the appropriate bins, and then just pulling up the points I need for an interpolation by looking at the neighboring bins. It may be a lot of coding, but it will be reasonably fast and use less memory than the Delaunay


Instruction: RuntimeError: stack expects each tensor to be equal size

Input:
I apologize in advance in this was asked before. I genuinely did not understand the solution.
MAX_LEN = 160
BATCH_SIZE = 16
EPOCHS = 10

class GPReviewDataset(data.Dataset):
    def __init__(self, review, target, tokenizer, max_len):
        self.review = review
        self.target = target
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.review)

    def __getitem__(self, item):
        review = str(self.review[item])

        encoding = tokenizer.encode_plus(text=review,
                                         max_length=self.max_len,
                                         add_special_tokens=True, padding='max_length',
                                         return_attention_mask=True,
                                         return_token_type_ids=False, return_tensors='pt')

        return {'review': review,
                'input_ids': encoding['input_ids'].flatten(),
                'attention_mask': encoding['attention_mask'].flatten(),
                'targets': torch.tensor(self.target[item], dtype=torch.long)}

free_df_train, free_df_test = train_test_split(free_df, test_size=0.2)
free_df_val, free_df_test = train_test_split(free_df_test, test_size=0.5)

def create_data_loader(df, tokenizer, max_len, batch_size):
    ds = GPReviewDataset(review=df.content.to_numpy(),
                         target=df['score'].to_numpy(),
                         tokenizer=tokenizer,
                         max_len=max_len)

    return data.DataLoader(ds, batch_size=batch_size,
                           num_workers=0)

train_data_loader = create_data_loader(free_df_train, tokenizer, MAX_LEN, BATCH_SIZE)
val_data_loader = create_data_loader(free_df_val, tokenizer, MAX_LEN, BATCH_SIZE)
test_data_loader = create_data_loader(free_df_test, tokenizer, MAX_LEN, BATCH_SIZE)

data = next(iter(train_data_loader))

I wrote a function later on to take in train_data_loader when training the data, but it was giving me the Runtime error. It seems like the proper solution is to use some sort of collate_fn; however I am confused on how exactly to apply that function.
My error below:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<timed exec> in <module>

<ipython-input-26-8ba1e19dd195> in train_epoch(model, data_loader, loss_fn, optimizer, device, scheduler, n_examples)
      4     correct_predictions = 0
      5
----> 6     for i in data_loader:
      7         input_ids = i['input_ids'].to(device)
      8         attention_mask = i['attention_mask'].to(device)

~\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    361
    362     def __next__(self):
--> 363         data = self._next_data()
    364         self._num_yielded += 1
    365         if self._dataset_kind == _DatasetKind.Iterable and \

~\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    401     def _next_data(self):
    402         index = self._next_index()  # may raise StopIteration
--> 403         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    404         if self._pin_memory:
    405             data = _utils.pin_memory.pin_memory(data)

~\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     45         else:
     46             data = self.dataset[possibly_batched_index]
---> 47         return self.collate_fn(data)

~\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     72         return batch
     73     elif isinstance(elem, container_abcs.Mapping):
---> 74         return {key: default_collate([d[key] for d in batch]) for key in elem}
     75     elif isinstance(elem, tuple) and hasattr(elem, '_fields'):  # namedtuple
     76         return elem_type(*(default_collate(samples) for samples in zip(*batch)))

~\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py in <dictcomp>(.0)
     72         return batch
     73     elif isinstance(elem, container_abcs.Mapping):
---> 74         return {key: default_collate([d[key] for d in batch]) for key in elem}
     75     elif isinstance(elem, tuple) and hasattr(elem, '_fields'):  # namedtuple
     76         return elem_type(*(default_collate(samples) for samples in zip(*batch)))

~\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     53             storage = elem.storage()._new_shared(numel)
     54             out = elem.new(storage)
---> 55         return torch.stack(batch, 0, out=out)
     56     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
     57             and elem_type.__name__ != 'string_':

RuntimeError: stack expects each tensor to be equal size, but got [160] at entry 0 and [376] at entry 5

Output: try this instead add (padding = 'max_length') in encode_plus


Instruction: PyTorch how to do gathers over multiple dimensions

Input:
I'm trying to find a way to do this without for loops.
Say I have a multi-dimensional tensor t0:
bs = 4
seq = 10
v = 16
t0 = torch.rand((bs, seq, v))

This has shape: torch.Size([4, 10, 16])
I have another tensor labels that is a batch of 5 random indices in the seq dimension:
labels = torch.randint(0, seq, size=[bs, sample])

So this has shape torch.Size([4, 5]). This is used to index the seq dimension of t0.
What I want to do is loop over the batch dimension doing gathers using labels tensor. My brute force solution is this:
t1 = torch.empty((bs, sample, v))
for b in range(bs):
    for idx0, idx1 in enumerate(labels[b]):
        t1[b, idx0, :] = t0[b, idx1, :]

Resulting in tensor t1 which has shape: torch.Size([4, 5, 16])
Is there a more idiomatic way of doing this in pytorch?

Output:
You can use fancy indexing here to select the desired portion of the tensor.
Essentially, if you generate the index arrays conveying your access pattern beforehand, you can directly use them to extract some slice of the tensor. The shape of the index arrays for each dimension should be same as that of the output tensor or slice you want to extract.
i = torch.arange(bs).reshape(bs, 1, 1) # shape = [bs, 1,      1]
j = labels.reshape(bs, sample, 1)      # shape = [bs, sample, 1]
k = torch.arange(v)                    # shape = [v, ]

# Get result as
t1 = t0[i, j, k]

Note the shapes of the above 3 tensors. Broadcasting appends extra dimesions in the front of a tensor, thus essentially reshaping k to [1, 1, v] shape which makes all 3 of them compatible for elementwise operations.
After broadcasting (i, j, k) together will produce 3 [bs, sample, v] shaped arrays and those will (elementwise) index your original tensor to produce the output tensor t1 of shape [bs, sample, v].


Instruction: Is there a way to classify a set of data as a whole via Pytorch?

Input:
I'm currently dealing with a classification task on a CT dataset. In CT datasets, multiple slices belong to one single patient, while setting up my dataset, I arrange my data as follows:
dataset/0/patient_1/1.png,2.png...
dataset/0/patient_2/1.png,2.png...
I wonder is there a way to let my network to classify by patient instead of by slices?
thank you

Output:
Each slice is a 2D image, while for each patient you have a 3D volume of CT voxels.
If you want to work per-patient, rather than per-slice, you'll need to organize your data to output batches of 3D information (of shape batchxchannelxdepthxheightxwidth) and make your model process 3D information (e.g., using Conv3D instead of Conv2D)


Instruction: PyTorch GPU memory leak during inference

Input:
I am trying to encode documents sentence-wise with a huggingface transformer module. I'm using the very small google/bert_uncased_L-2_H-128_A-2 pretrained model with the following code:
def pre_encode_wikipedia(model, tokenizer, device, save_path):

  document_data_list = []

  for iteration, document in enumerate(wikipedia_small['text']):
    torch.cuda.empty_cache()

    sentence_embeds_per_doc = [torch.randn(128)]
    attention_mask_per_doc = [1]
    special_tokens_per_doc = [1]

    doc_split = nltk.sent_tokenize(document)
    doc_tokenized = tokenizer.batch_encode_plus(doc_split, padding='longest', truncation=True, max_length=512, return_tensors='pt')

    for key, value in doc_tokenized.items():
      doc_tokenized[key] = doc_tokenized[key].to(device)

    with torch.no_grad():
      doc_encoded = model(**doc_tokenized)

    for sentence in doc_encoded['last_hidden_state']:
      sentence[0].to('cpu')
      sentence_embeds_per_doc.append(sentence[0])
      attention_mask_per_doc.append(1)
      special_tokens_per_doc.append(0)

    sentence_embeds = torch.stack(sentence_embeds_per_doc)
    attention_mask = torch.FloatTensor(attention_mask_per_doc)
    special_tokens_mask = torch.FloatTensor(special_tokens_per_doc)

    document_data = torch.utils.data.TensorDataset(*[sentence_embeds, attention_mask, special_tokens_mask])
    torch.save(document_data, f'{save_path}{time.strftime("%Y%m%d-%H%M%S")}{iteration}.pt')
    print(f"Document at {iteration} encoded and saved.")

After about 200-300 iterations on my local GTX 1060 3GB I get an error saying that my CUDA memory is full. Running this code on Colab with more GPU RAM gives me a few thousand iterations.
Things I've tried:

Adding torch.cuda.empty_cache() to the start of every iteration to clear out previously held tensors
Wrapping the model in torch.no_grad() to disable the computation graph
Setting model.eval() to disable any stochastic properties that might take up memory
Sending the output straight to CPU in hopes to free up memory

I'm baffled as to why my memory keeps overflowing. I've trained several models of bigger sizes, applying all the standard practices of a training loop (optimizer.zero_grad(), etc.) I've never had this problem. Why does it appear during this seemingly trivial task?
Edit #1
Changing sentence[0].to('cpu') to cpu_sentence = sentence[0].to('cpu') gave me a few thousand iterations before VRAM usage suddenly spiked, causing the run to crash:

Output:
Can you try replacing
sentence[0].to('cpu')

with
cpu_sentence = sentence[0].to('cpu')

See more info here https://pytorch.org/docs/stable/tensors.html#torch.Tensor.to


Instruction: ModuleNotFoundError: No module named 'torch.nn'; 'torch' is not a package on Mac OS

Input:
I am trying to get pytorch to work but I keep getting this error.
ModuleNotFoundError: No module named 'torch.nn'; 'torch' is not a package
I am using a Macbook, i've tried looking at the other answers on here but nothing is working.
import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

Output:
Maybe you can check conda list to see if there is PyTorch installed. You should be able to run torch if you had installed PyTorch.
Download link: https://pytorch.org/get-started/locally/
Just remember to install CUDA additionally if you want to use GPU instead of CPU.


Instruction: PyTorch RuntimeError: Tensor for argument #1 'self' is on CPU, but expected them to be on GPU

Input:
I'm using PyTorch for my Logistic Regression model but whenever I run the model summary I get an error

RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

Code
# Convert data to tensors
X_train = torch.Tensor(X_train)
y_train = torch.LongTensor(y_train)
X_test = torch.Tensor(X_test)
y_test = torch.LongTensor(y_test)

class LogisticRegression(nn.Module):
    def __init__(self, input_features, num_classes):
        super(LogisticRegression, self).__init__()
        self.fc1 = nn.Linear(input_dim, num_classes)

    def forward(self, x_in, apply_softmax = False):
        y_pred = self.fc1(x_in)
        if apply_softmax:
            y_pred = F.softmax(y_pred, dim = 1)
        return y_pred

INPUT_DIM = X_train.shape[1]
NUM_CLASSES = len(y_train.unique())

model = LogisticRegression(input_features = INPUT_DIM, num_classes = NUM_CLASSES)
print(model.named_parameters)
summary(model, input_size=(INPUT_DIM,))

My way does not work as expected, how do I go about fixing the problem?

Output:
I had the same error.

RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

Ensuring the model and its weights were on the GPU helped:
model.to(device)

where device is defined:
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using {device} device")


Instruction: Pytorch already installed using Conda but fails when called

Input:
I am trying to install pytorch for using BERT but when following the installation instructions found here: https://pytorch.org/get-started/locally/ I am getting an error.
When I try and initalise the BERT model I get the following error:
ImportError:
    BertForSequenceClassification requires the PyTorch library but it was not found in your environment.
    Checkout the instructions on theinstallation page: https://pytorch.org/get-started/locally/
    and follow the ones that match your environment.

I have followed the instructions and run the following command line in my Conda prompt terminal AND in my current working directory:
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch

It looks to complete but when I try and call the following line I get the same error as the start as if it hasn't installed at all.
Can anyone help me out please.
EDIT:
The code I am using to execute bert is:
model = BertForSequenceClassification.from_pretrained(r'C:\Users\441\bert\pytorch_model.bin', config = r'C:\Users\441\bert\config.json')

Output: I had the same issue (same error msg), and after using conda list | grep torch I also found it is there. What worked for me is that I restarted the jupyter notebook kernel and the error is gone.


Instruction: Torch model forward with a diferent image size

Input:
I am testing some well known models for computer vision: UNet, FC-DenseNet103, this implementation
I train them with 224x224 randomly cropped patches and do the same on the validation set.
Now when I run inference on some videos, I pass it the frames directly (1280x640) and it works. It runs the same operations on different image sizes and never gives an error. It actually gives a nice output, but the quality of the output depends on the image size...
Now it's been a long time since I've worked with neural nets but when I was using tensorflow I remember I had to crop the input images to the train crop size.
Why don't I need to do this anymore? What's happening under the hood?

Output: It seems that the models that you are using have no linear layers. Because of this the output of the convolutional layers go straight into the softmax function. The softmax function doesn't take a specific shape for its input so it can take any shape as input. Because of this your model will work with any shape of image but the accuracy of your model will probably be far worse given different image shapes than the one you trained on.


Instruction: all pairwise dot product pytorch

Input:
Is there a built in function to calculate efficiently all pairwaise dot products of two tensors in Pytorch?
e.g.
input - tensor A (shape NxD)
tensor B (shape NxD)
output - tensor C (shape NxN) such that C_i,j = torch.dot(A_i, B_j) ?

Output:
Isn't it simply
C = torch.mm(A, B.T)  # same as C = A @ B.T

BTW,
A very flexible tool for matrix/vector/tensor dot products is torch.einsum:
C = torch.einsum('id,jd->ij', A, B)


Instruction: Is there a way to figure out whether PyTorch model is on cpu or on the device?

Input:
I would like to figure out, whether the PyTorch model is on cpu or cuda in order to
initialize some other variable as Torch.Tensor or Torch.cuda.Tensor depending on the model.
However, looking at the output of the dir() function I see only .cpu(), .cuda(), to() methods which put the model on device, GPU or other device, specified in to. For PyTorch tensor there is is_cuda attribute, but no analogue for the whole model.
Is there some way to deduce this for a model, or one needs to refer to a particular weight?

Output:
No, there is no such function for nn.Module, I believe this is because parameters could be on multiple devices at the same time.
If you're working with a single device, a workaround is to check the first parameter:
next(model.parameters()).is_cuda

As described here.


Instruction: diffrence between a[:,:,0] and a[:][:][0]

Input:
Hi I was studying slicing in python and I found something strange and I don't understand
import torch

a = torch.tensor([
    [
        [1, 2, 3],
        [4, 5, 6]
    ],
    [
        [7, 2, 3],
        [8, 5, 6]
    ]
])

>>> a[:][:][0]
tensor([[1, 2, 3],
        [4, 5, 6]])

>>> a[:,:,0]
tensor([[1, 4],
        [7, 8]])

I tried to pull out [[1,4,7,8]] from the corresponding torch list, so I entered a[:][:][0] and the result of it is [[1,2,3], [4,5,6]]. Then, when I input a[:,:,0], then [[1,4,7,8]] appeared.
I thought they have no different but different result appeared.
the torch and numpy operators, there was an operation like a[:,0]. How exactly can it be different from a[:][0]?

Output:
You can see the first, a[:][:][0], as several, chained calls to __getitem__. That means a[:][:][0] is roughly equivalent to this:
b = a[:]
c = b[:]
d = c[0]

Where d is the result. In your case, it returns the same thing as a[0], because a[:] == a.
In contrast, a[:,:,0] will only call __getitem__ once with parameters slice(None), slice(None), 0.
In your case, that's the first slice of your tensor on the third axis.


Instruction: How can I solve "torch.utils.ffi is deprecated. Please use cpp extensions instead" without downgrade pytorch version?

Input:
When I run the code below it shows me the error.
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.
I have been searching solution on the online. The problem is the code below working in old version of torch (0.4.1). I want to know whether it is possible to modify or replace this code for working in the new version of pytorch.
from torch.utils.ffi import _wrap_function
from ._nms import lib as _lib, ffi as _ffi

__all__ = []
def _import_symbols(locals):
    for symbol in dir(_lib):
        fn = getattr(_lib, symbol)
        if callable(fn):
            locals[symbol] = _wrap_function(fn, _ffi)
        else:
            locals[symbol] = fn
        __all__.append(symbol)

_import_symbols(locals())

Output:
I am facing the same problem have just seen some useful information in:

https://pytorch.org/tutorials/advanced/cpp_extension.html
https://pytorch.org/docs/stable/cpp_extension.html

To avoid downgrade the version of PyTorch, you should consider to use the following libraries while finding more details in the above links:
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CppExtension


Instruction: FastAi Learner changes the dataloader Why do they do it? How is this the right thing to do?

Input:
The code below shows that passing a dataloader to a learner changes it. This seems like a very odd behavior. Why is it done this way, what is the logic for the change and how can I turn it off?
More importantly, the dataloader also has the val and test data in it. If the learner goes around changing it then it should be very well documented on what its doing. Nothing is mentioned about changing the data loader in cnn_learner and Learner.
dls = ImageDataLoaders.from_name_func(path, get_image_files(path), valid_pct=0.2,
label_func=is_tb,item_tfms=Resize(224))

x,y=next(iter(dls[0]))
print(x.min(),x.max())

This gives 0 and 1 respectively. However, initiating a learner

learn = cnn_learner(dls, resnet34, metrics=[accuracy],n_out=2,loss_func=CrossEntropyLossFlat())

x,y=next(iter(dls[0]))
print(x.min(),x.max())

I get -2.11 and 2.64 respectively.

Output:
The ImageDataLoaders.from_name_func dataloader shuffles the dataset by default.
You can pass it shuffle_train=False if you don't want to.


Instruction: Get hash value of a pytorch architecture?

Input:
I would like to automatically check whether a certain architecture has already been trained on a task. My thought is: If I can get a hash value of the architecture and store this value in a .json file, then I can check whether it has already been trained by checking whether the architecture's hash value is in the .json file.
However, I'm not exactly sure what to hash: If I hash the module object, then it will be different each time I run the program since it'll have a different id, because it's in a different memory location. Also, different random initializations will probably cause a different hash value.
Is there a way I can get a hash value that will be the same so long as the module consists of the same layers with the same dimensions?

Output:
I think hashing the string representation of the model might be a solution to your problem.
hash(str(model))


Instruction: PyTorch - Creating Federated CIFAR-10 Dataset

Input:
I'm training a neural network (doesn't matter which one) on CIFAR-10 dataset. I'm using Federated Learning:

I have 10 models, each model having access to its own part of the dataset. At every time step, each model makes a step using its own data, and then the global model is an average of the model (this version is based on this, but I tried a lot of options):

def server_aggregate(server_model, client_models):
    global_dict = server_model.state_dict()
    for k in global_dict.keys():
        global_dict[k] = torch.stack([client_models[i].state_dict()[k].float() for i in range(len(client_models))], 0).mean(0)
    server_model.load_state_dict(global_dict)
    for model in client_models:
        model.load_state_dict(server_model.state_dict())

To be specific, each machine only has access to a data corresponding to a single class. I.e. machine 0 has only samples corresponding to class 0, etc. I'm doing it the following way:

def split_into_classes(full_ds, batch_size, num_classes=10):
  class2indices = [[] for _ in range(num_classes)]
  for i, y in enumerate(full_ds.targets):
    class2indices[y].append(i)

  datasets = [torch.utils.data.Subset(full_ds, indices) for indices in class2indices]
  return [DataLoader(ds, batch_size=batch_size, shuffle=True) for ds in datasets]

Problem. During training, I can see that my federated training loss decreases. However, I never see my test loss/accuracy improve (acc is always around 10%).
Moreover, when I check accuracy on train/test datasets:

For the federated dataset, the accuracy improves.
For the testing dataset, the accuracy doesn't improve.
(Most surprising) for the training dataset, the accuracy doesn't improve. Note that this dataset is essentially the same as federated dataset, but not split into classes. The checking code is the following:

def epoch_summary(model, fed_loaders, true_train_loader, test_loader, frac):
  with torch.no_grad():
    train_len = 0
    train_loss, train_acc = 0, 0
    for train_loader in fed_loaders:
      cur_loss, cur_acc, cur_len = true_results(model, train_loader, frac)
      train_loss += cur_len * cur_loss
      train_acc += cur_len * cur_acc
      train_len += cur_len

    train_loss /= train_len
    train_acc /= train_len

    true_train_loss, true_train_acc, true_train_len = true_results(model, true_train_loader, frac)
    test_loss, test_acc, test_len = true_results(model, test_loader, frac)

  print("TrainLoss: {:.4f} TrainAcc: {:.2f} TrueLoss: {:.4f} TrueAcc: {:.2f} TestLoss: {:.4f} TestAcc: {:.2f}".format(
        train_loss, train_acc, true_train_loss, true_train_acc, test_loss, test_acc
        ), flush=True)

The full code can be found here. Things which don't seem to matter:

Model. I got the same problem for Resnet models and for some other models.
How I aggregate the models. I tried using state_dict or directly manipulate model.parameters(), no effect.
How I learn the models. I tried using optim.SGD or directly update param.data -= learning_rate * param.grad, no effect.
Computational graph. I've tried adding .detach().clone() and with torch.no_grad() into all possible places, no effect.

So I'm suspecting that the problem is somehow with the federated data itself (especially given strange accuracy results). What can be a problem?

Output:
10% on CIFAR-10 is basically random - your model outputs labels at random and gets 10%.
I think the problem lies in your "federated training" strategy: you cannot expect your sub-models to learn anything meaningful when all they see is a single label. This is why training data is shuffled.
Think of it: if each of your sub models learns all weights to be zero apart from the bias vector of the last classification layer that has 1 in the entry corresponding to the class this sub-model sees - the training of each sub model is perfect (it gets it right for all training samples it sees), but the averaged model is meaningless.


Instruction: Can I access the inner layer outputs of DeepLab in pytorch?

Input:
Using Pytorch, I am trying to implement a network that is using the pre=trained DeepLab ResNet-101.
I found two possible methods for using this network:
this one
or
 torchvision.models.segmentation.deeplabv3_resnet101(
     pretrained=False, progress=True, num_classes=21, aux_loss=None, **kwargs)

However, I might not only need this network's output, but also several inside layers' outputs.
Is there a way to access the inner layer outputs using one of these methods?
If not - Is it possible to manually copy the trained resnet's parameters so I can manually recreate it and add those outputs myself? (Hopefully the first option is possible so I won't need to do this)
Thanks!

Output:
You can achieve this without too much trouble using forward hooks.
The idea is to loop over the modules of your model, find the layers you're interested in, hook a callback function onto them. When called, those layers will trigger the hook. We will take advantage of this to save the intermediate outputs.
For example, let's say you want to get the outputs of layer classifier.0.convs.3.1:
layers = ['classifier.0.convs.3.1']
activations = {}

def forward_hook(name):
    def hook(module, x, y):
        activations[name] = y
    return hook

for name, module in model.named_modules():
    if name in layers:
        module.register_forward_hook(forward_hook(name))

*The closure around hook() made by forward_hook's scope is used to enclose the module's name which you wouldn't otherwise have access to at this point.
Everything is ready, we can call the model
>>> model = torchvision.models.segmentation.deeplabv3_resnet101(
        pretrained=True, progress=True, num_classes=21, aux_loss=None)

>>> model(torch.rand(16, 3, 100, 100))

And as expected, after inference, activations will have a new entry 'classifier.0.convs.3.1' which - in this case - will contain a tensor of shape (16, 256, 13, 13).

Not so long ago, I wrote an answer about a similar question which goes a little bit more in detail on how hooks can be used to inspect the intermediate output shapes.