Instruction: Is there a way to monitor optimizer's step in Pytorch?

Input:
Consider that you are using a Pytorch optimizer such as torch.optim.Adam(model_parameters).

So in your training loop you will have something like:

optimizer = torch.optim.Adam(model_parameters)
# put the training loop here

loss.backward()
optimizer.step()
optimizer.zero()

Is there a way to monitor what steps are taking your optimizer ? To make sure that you are not on a flat area and thus taking no steps since the gradient are null. Maybe checking the learning rate would be a solution ?

Output:
Answering to myself here.

The best practice would be (in PyTorch) to check the gradient of the leaf tensors.
If the gradient is None but is_leaf attribute is set to True, something is obviously buggy.

torch.nn.Parameters('insert your tensor here') is especially confusing to this regard.
As the Tensor needs to be defined as a torch.nn.Parameters to be succesfully updated.
I would advise not to use tensor.requires_grad_(True) as is confuses torch. Only set your parameters as above.


Instruction: Custom dataset os.path.join () returns type error

Input:
I am writing custom dataset but it returns type error when I am merging root directory path with pandas iloc of image name in csv file:

 img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])

error:  TypeError: join() argument must be str or bytes, not 'int64'

I have tried converting the annotation.iloc to string type but it's still giving me the same error.

csvfile with filenames and labels:

custom dataset class:

class patientdataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
            self.annotations = pd.read_csv(csv_file)
            self.root_dir = root_dir
            self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
        image= np.array(np.load(img_path))
        y_label = torch.tensor(self.annotations.iloc[index, 1]).long()

        if self.transform:
            imagearrays = self.transform(image)
            image = imagearrays[None, :, :, :]
            imaget = np.transpose(image, (0, 2, 1, 3))
            image = imaget

        return (image, y_label)

Output:
According to your dataset (attached csv file), pd.read_csv(csv_file) produces dataframe with 3 columns: 1st for index, 2nd for filename, and 3rd for label. And this img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0]) line is not working because iloc[index, 0] is about the 1st column, it will extract index data and not a file name, and join expects to get 2 strings, that's why you are getting TypeError.

Based on your csv file example, you should do:

class patientdataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
            self.annotations = pd.read_csv(csv_file)
            self.root_dir = root_dir
            self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 1])  # 1 - for file name (2nd column)
        image= np.array(np.load(img_path))
        y_label = torch.tensor(self.annotations.iloc[index, 2]).long()  # 2 - for label (3rd column)

        if self.transform:
            imagearrays = self.transform(image)
            image = imagearrays[None, :, :, :]
            imaget = np.transpose(image, (0, 2, 1, 3))
            image = imaget

        return (image, y_label)


Instruction: problem on training(first epoch take a long time more than 4 hours )using colab and fastai

Input:
I have a problem on training (first epoch takes a long time more than 4 hours) using colab and fastai. It takes me about 5 hours only for the first epoch which is not useful using google colab because the limitation using of GPU
I'm using  'efficientnet-b4' and about 52k training photos after data augmentation and 11k valid photos.

The problem is shown on the photo:

Output: Colab takes alot of time fetching the data from google drive as it is not indexed efficiently and thus makes the first epoch very long. For large datasets, I would recommend using google cloud storage and importing data from there.


Instruction: PyInstaller executable fails to get source code of TorchScript

Input:
I'm trying to make Windows executable of my script that includes PyTorch. My script's imports are:

import numpy.core.multiarray  # which is a workaround for "ImportError: numpy.core.multiarray failed to import"
import six # which is workaround for "ModuleNotFoundError: No module named 'six'"
import torch
import torch.nn as nn
import warnings
import argparse
import json
import math
import numpy as np
import jsonschema
import os
from datetime import datetime
from sklearn.mixture import GaussianMixture
from scipy.io import wavfile
from scipy.signal import get_window
from scipy.signal import spectrogram

I'm using command:

pyinstaller --hidden-import pkg_resources.py2_warn extractor.py

PyInstaller throws no error while creating .exe, but when I run the .exe i get:

Traceback (most recent call last):
  File "site-packages\torch\_utils_internal.py", line 46, in get_source_lines_and_file
  File "inspect.py", line 967, in getsourcelines
  File "inspect.py", line 798, in findsource
OSError: could not get source code

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "extractor.py", line 3, in <module>
    import torch
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\ProgramData\Anaconda3\envs\forexe2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\torch\__init__.py", line 367, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\ProgramData\Anaconda3\envs\forexe2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\torch\distributions\__init__.py", line 112, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\ProgramData\Anaconda3\envs\forexe2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\torch\distributions\von_mises.py", line 55, in <module>
  File "site-packages\torch\jit\__init__.py", line 1287, in script
  File "site-packages\torch\jit\frontend.py", line 164, in get_jit_def
  File "site-packages\torch\_utils_internal.py", line 53, in get_source_lines_and_file
OSError: Can't get source for <function _rejection_sample at 0x0000000006892F70>. TorchScript requires source access in order to carry out compilation, make sure original .py files are available. Original error: could not get source code
[5704] Failed to execute script extractor

Which I don't understand. This may be similar issue to this question. What is causing the problem?

I'm using conda env, with torch installed via pip (which is a workaround for the torch to be correctly hooked).

Windows 10
Python 3.8.2
torch 1.5.0+cu101
torchvision 0.6.0+cu101 (also tried with 0.2.2)
PyInstaller 3.6

Output:
Torch is open source, so you can search for the function _rejection_sample on the torch GitHub. This identifies the problematic file as torch.distributions.von_mises. If your program is not using the torch.distributions module, you can simply exclude it by changing the .spec file generated by pyinstaller.

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None
excluded_modules = ['torch.distributions'] # <<< ADD THIS LINE

a = Analysis(['C:/your/path/here'],
             pathex=['C:\\your\\path\\here'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=excluded_modules,    # <<< CHANGE THIS LINE
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
# remaining code omitted for brevity

You should only need to make changes in the two places specified above. The rest should already be there.

Then, build from the .spec file using

pyinstaller your_file.spec

In the future, consider using debug flags when you build your project. This also identifies the location of the file causing problems.


Instruction: What does *variable.shape mean in python

Input:
I know "*variable_name" assists in packing and unpacking.

But how does variable_name.shape work? Unable to visualize why the second dimension is squeezed out when prefixing with ""?

print("top_class.shape {}".format(top_class.shape))
top_class.shape torch.Size([64, 1])

print("*top_class.shape {}".format(*top_class.shape))
*top_class.shape 64

Output:
for numpy.array that is extensively used in math-related and image processing programs, .shape describes the size of the array for all existing dimensions:

>>> import numpy as np
>>> a = np.zeros((3,3,3))
>>> a
array([[[ 0.,  0.,  0.],
        [ 0.,  0.,  0.],
        [ 0.,  0.,  0.]],

       [[ 0.,  0.,  0.],
        [ 0.,  0.,  0.],
        [ 0.,  0.,  0.]],

       [[ 0.,  0.,  0.],
        [ 0.,  0.,  0.],
        [ 0.,  0.,  0.]]])
>>> a.shape
(3, 3, 3)
>>>

The asterisk "unpacks" the tuple into several separate arguments, in your case (64,1) becomes 64, 1, so only the first one get printed because there's only one format specification.


Instruction: How to calculate roc auc score for the whole epoch like avg accuracy?

Input:
I am implementing a training loop in PyTorch and for metrics, I want to use ROC AUC score using sklearn.metrics.roc_auc_score.

I can use sklearn's implementation for calculating the score for a single prediction but have a little trouble imagining how to use it to calculate the average score for the whole epoch. Can anyone push me in the right direction?

Output: y_true and y_score, in the function can be 1-D arrays, so if you collect the values form the entire epoch, you can directly call the function. Note that if you do multi-label classification, you need to compute the ROC AUC score for each class separately.


Instruction: How can I set maximum n elements in one row in pytorch tensor to 1 and оther to zero?

Input:
For example, if I have tensor

0.8, 0.1, 0.9, 0.2
0.7, 0.1, 0.4, 0.6

and n = 2, I want to get

1, 0, 1, 0
1, 0, 0, 1

Or maybe it's better to do it numpy, but the question is the same.

Output:
For performance efficiency, we can use np.argpartition -

def partition_assign(a, n):
    idx = np.argpartition(a,-n,axis=1)[:,-n:]
    out = np.zeros(a.shape, dtype=int)
    np.put_along_axis(out,idx,1,axis=1)
    return out

Or we can use np.argpartition(-a,n,axis=1)[:,:n] at that argpartition step.

Sample runs -

In [56]: a
Out[56]:
array([[0.8, 0.1, 0.9, 0.2],
       [0.7, 0.1, 0.4, 0.6]])

In [57]: partition_assign(a, n=2)
Out[57]:
array([[1, 0, 1, 0],
       [1, 0, 0, 1]])

In [58]: partition_assign(a, n=3)
Out[58]:
array([[1, 0, 1, 1],
       [1, 0, 1, 1]])


Instruction: Add multiple tensors inplace in PyTorch

Input:
I can add two tensors x and y inplace like this

x = x.add(y)

Is there a way of doing the same with three or more tensors given all tensors have same dimensions?

Output:
result = torch.sum(torch.stack([x, y, ...]), dim=0)

Without stack:
from functools import reduce

result = reduce(torch.add, [x, y, ...])

EDIT
As @LudvigH pointed out, the second method is not as memory-efficient, as inplace addition. So it's better like this:
from functools import reduce

result = reduce(
    torch.Tensor.add_,
    [x, y, ...],
    torch.zeros_like(x)  # optionally set initial element to avoid changing `x`
)


Instruction: Converting two Numpy data sets into a particularr PyTorch data set

Input:
I want to play around with a neural network that recognizes handwritten numbers. I found some of these on the web which use PyTorch, however they seem to download the data from the MNIST website in a particular format. My data is, however, available as follows:

with np.load('prediction-challenge-01-data.npz') as fh:
     data_x = fh['data_x']
     data_y = fh['data_y']

Where data_x is the training data and data_y are the labels of the pictures. I want these data sets to be in the same format as trainloader as shown below:

trainset = datasets.MNIST('/data/mnist', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

Where trainloader already has the training set data_x and labels data_y together in one set.

Is there any way to do this?

Edit: Shapes of data_x and data_y:

In [1]:  data_x.shape
Out[2]: (20000, 1, 28, 28)

In [5]:  data_y.shape
Out[7]: (20000,)

Output:
You can easily create your own dataset. Just inherit from torch.utils.data.Dataset and implement
__getitem__ at the very least:
Here is a quick and dirty example to get you going:

class YourOwnDataset(torch.utils.data.Dataset):
    def __init__(self, input_file_path, transformations) :
        super().__init__()
        self.path = input_file_path
        self.transforms = transformations

        with np.load(self.path) as fh:
            # I assume fh['data_x'] is a list you get the idea
            self.data = fh['data_x']
            self.labels = fh['data_y']

    # in getitem, we retrieve one item based on the input index
    def __getitem__(self, index):
        data = self.data[index]
        # based on the loss you chose and what you have in mind,
        # you can transform you label, here I assume they are
        # integer numbers (like, 1, 3, etc as labels used for classification)
        label = self.labels[index]
        img = convert/reshape your data into img
        img = self.transforms(img)
        return img, labels

    def __len__(self):
        return len(self.data)

and you can create your dataset like :

from torchvision import transforms

# add any number of transformations you like, I just added ToTensor()
transformations = transforms.Compose([transforms.ToTensor()])

trainset = YourOwnDataset('prediction-challenge-01-data.npz', transformations )
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)


Instruction: RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Input:
I set my model and data to the same device, but always raise the error like this:

  RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

The following is my training code, I hope you can answer it.Thanks!

def train(train_img_path, train_label_path, pths_path, interval, log_file):

    file_num = len(os.listdir(train_img_path))
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    net = EAST(extractor=extractor, geometry_mode=geometry_mode, pretrained=True)
    net = net.to(device)
    trainset = custom_dataset(train_img_path, train_label_path)
    train_loader = data.DataLoader(trainset, batch_size=batch_size,
                                   shuffle=True, num_workers=num_workers, drop_last=True)

    optimizer = optim.SGD(net.parameters(), lr=initial_lr, momentum=momentum, weight_decay=weight_decay_sgd)
    criterion = Loss(weight_geo, weight_angle, geometry_mode="RBOX")

    net.train()
    epoch_loss = 0.
    for epoch in range(max_epoch):
        epoch_time = time.time()
        for i, (img, score_gt, geo_gt, ignored_map) in enumerate(train_loader):
            start_time = time.time()
            img, score_gt, geo_gt, ignored_map = img.to(device), score_gt.to(device),\
                                                 geo_gt.to(device), ignored_map.to(device)

            score_pred, geo_pred = net(img)
            total_loss, score_loss, loss_AABB, loss_angle = criterion(score_pred, geo_pred, score_gt, geo_gt, ignored_map)

            epoch_loss += total_loss.item()

            optimizer.zero_grad()
            total_loss.backward()

            optimizer.step()

Output:
I suspect your loss function has some internal parameters of its own, therefore you should also

criterion = Loss(weight_geo, weight_angle, geometry_mode="RBOX").to(device)

It would be easier to spot the error if you provide a full trace, indicating which line exactly caused the error.


Instruction: Training loss not changing at all (PyTorch)

Input:
I am trying to solve a text classification problem. My training data has input as a sequence of 80 numbers in which each represent a word and target value is just a number between 1 and 3.
I pass it through this model:

class Model(nn.Module):
    def __init__(self, tokenize_vocab_count):
        super().__init__()
        self.embd = nn.Embedding(tokenize_vocab_count+1, 300)
        self.embd_dropout = nn.Dropout(0.3)
        self.LSTM = nn.LSTM(input_size=300, hidden_size=100, dropout=0.3, batch_first=True)
        self.lin1 = nn.Linear(100, 1024)
        self.lin2 = nn.Linear(1024, 512)
        self.lin_dropout = nn.Dropout(0.8)
        self.lin3 = nn.Linear(512, 3)

    def forward(self, inp):
        inp = self.embd_dropout(self.embd(inp))
        inp, (h_t, h_o) = self.LSTM(inp)
        h_t = F.relu(self.lin_dropout(self.lin1(h_t)))
        h_t = F.relu(self.lin_dropout(self.lin2(h_t)))
        out = F.softmax(self.lin3(h_t))
        return out

My training loop is as follows:

model = Model(tokenizer_obj.count+1).to('cuda')

optimizer = optim.AdamW(model.parameters(), lr=1e-2)
loss_fn = nn.CrossEntropyLoss()

EPOCH = 10

for epoch in range(0, EPOCH):
     for feature, target in tqdm(author_dataloader):
         train_loss = loss_fn(model(feature.to('cuda')).view(-1,  3), target.to('cuda'))
         optimizer.zero_grad()
         train_loss.backward()
         optimizer.step()
      print(f"epoch: {epoch + 1}\tTrain Loss : {train_loss}")

I printed out the feature and target dimension and it is as follows:

torch.Size([64, 80]) torch.Size([64])

Here 64 is the batch_size.
I am not doing any validation as of now.
When I train I am getting a constant loss value and no change

/home/koushik/Software/miniconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/rnn.py:50: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.3 and num_layers=1
  "num_layers={}".format(dropout, num_layers))
  0%|                                                                                | 0/306 [00:00<?, ?it/s]/media/koushik/Backup Plus/Code/Machine Deep Learning/NLP/src/Deep Learning/model.py:20: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  out = F.softmax(self.lin3(h_t))
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 89.36it/s]
epoch: 1        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 89.97it/s]
epoch: 2        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 89.35it/s]
epoch: 3        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 89.17it/s]
epoch: 4        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 88.72it/s]
epoch: 5        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 87.75it/s]
epoch: 6        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 85.67it/s]
epoch: 7        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 85.40it/s]
epoch: 8        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 84.49it/s]
epoch: 9        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 84.21it/s]
epoch: 10       Train Loss : 1.0986120700836182

Can anyone please help

Output:
You're using nn.CrossEntropyLoss, which applies log-softmax, but you also apply softmax in the model:

out = F.softmax(self.lin3(h_t))

The output of your model should be the raw logits, without the F.softmax.


Instruction: How to implement tf.gather_nd in Pytorch with the argument batch_dims?

Input:
I have been doing a project on image matching, so I need to find correspondences between 2 images. To get descriptors, I will need a interpolate function. However, when I read about a equivalent function which is done in Tensorflow, I still don’t get how to implement tf.gather_nd(parmas, indices, barch_dims) in Pytorch. Especially when there is a argument: batch_dims. I have gone through stackoverflow and there is no perfect equivalence yet.

The referred interpolate function in Tensorflow is below and I have been trying to implement this in Pytorch Arguments' information is below:

inputs is a dense feature map[i] from a for loop of batch size, which means it is 3D[H, W, C](in pytorch is [C, H, W])

pos is a set of random point coordinate shapes like [[i, j], [i, j],...,[i, j]], so it is 2D when it goes in interpolate function(in pytorch is [[i,i,...,i], [j,j,...,j]])

and it then expands both of their dimensions when they get into this function

I just want a perfect implement of tf.gather_nd with argument batch_dims. Thank you!
And here's a simple example of using it:

pos = tf.ones((12, 2)) ## stands for a set of coordinates [[i, i,…, i], [j, j,…, j]]
inputs = tf.ones((4, 4, 128)) ## stands for [H, W, C] of dense feature map
outputs = interpolate(pos, inputs, batched=False)
print(outputs.get_shape()) # We get (12, 128) here

interpolate function (tf version):

def interpolate(pos, inputs, nd=True):

    pos = tf.expand_dims(pos, 0)
    inputs = tf.expand_dims(inputs, 0)

    h = tf.shape(inputs)[1]
    w = tf.shape(inputs)[2]

    i = pos[:, :, 0]
    j = pos[:, :, 1]

    i_top_left = tf.clip_by_value(tf.cast(tf.math.floor(i), tf.int32), 0, h - 1)
    j_top_left = tf.clip_by_value(tf.cast(tf.math.floor(j), tf.int32), 0, w - 1)

    i_top_right = tf.clip_by_value(tf.cast(tf.math.floor(i), tf.int32), 0, h - 1)
    j_top_right = tf.clip_by_value(tf.cast(tf.math.ceil(j), tf.int32), 0, w - 1)

    i_bottom_left = tf.clip_by_value(tf.cast(tf.math.ceil(i), tf.int32), 0, h - 1)
    j_bottom_left = tf.clip_by_value(tf.cast(tf.math.floor(j), tf.int32), 0, w - 1)

    i_bottom_right = tf.clip_by_value(tf.cast(tf.math.ceil(i), tf.int32), 0, h - 1)
    j_bottom_right = tf.clip_by_value(tf.cast(tf.math.ceil(j), tf.int32), 0, w - 1)

    dist_i_top_left = i - tf.cast(i_top_left, tf.float32)
    dist_j_top_left = j - tf.cast(j_top_left, tf.float32)
    w_top_left = (1 - dist_i_top_left) * (1 - dist_j_top_left)
    w_top_right = (1 - dist_i_top_left) * dist_j_top_left
    w_bottom_left = dist_i_top_left * (1 - dist_j_top_left)
    w_bottom_right = dist_i_top_left * dist_j_top_left

    if nd:
        w_top_left = w_top_left[..., None]
        w_top_right = w_top_right[..., None]
        w_bottom_left = w_bottom_left[..., None]
        w_bottom_right = w_bottom_right[..., None]

    interpolated_val = (
        w_top_left * tf.gather_nd(inputs, tf.stack([i_top_left, j_top_left], axis=-1), batch_dims=1) +
        w_top_right * tf.gather_nd(inputs, tf.stack([i_top_right, j_top_right], axis=-1), batch_dims=1) +
        w_bottom_left * tf.gather_nd(inputs, tf.stack([i_bottom_left, j_bottom_left], axis=-1), batch_dims=1) +
        w_bottom_right * tf.gather_nd(inputs, tf.stack([i_bottom_right, j_bottom_right], axis=-1), batch_dims=1)
    )

    interpolated_val = tf.squeeze(interpolated_val, axis=0)
    return interpolated_val

Output:
As far as I'm aware there is no directly equivalent of tf.gather_nd in PyTorch and implementing a generic version with batch_dims is not that simple. However, you likely don't need a generic version, and given the context of your interpolate function, a version for [C, H, W] would suffice.

At the beginning of interpolate you add a singular dimension to the front, which is the batch dimension. Setting batch_dims=1 in tf.gather_nd means there is one batch dimension at the beginning, therefore it applies it per batch, i.e. it indexes inputs[0] with pos[0] etc. There is no benefit of adding a singular batch dimension, because you could have just used the direct computation.

# Adding singular batch dimension
# Shape: [1, num_pos, 2]
pos = tf.expand_dims(pos, 0)
# Shape: [1, H, W, C]
inputs = tf.expand_dims(inputs, 0)

batched_result = tf.gather_nd(inputs, pos, batch_dims=1)
single_result = tf.gater_nd(inputs[0], pos[0])

# The first element in the batched result is the same as the single result
# Hence there is no benefit to adding a singular batch dimension.
tf.reduce_all(batched_result[0] == single_result) # => True

Single version

In PyTorch the implementation for [H, W, C] can be done with Python's indexing. While PyTorch usually uses [C, H, W] for images, it's only a matter of what dimension to index, but let's keep them the same as in TensorFlow for the sake of comparison. If you were to index them manually, you would do it as such: inputs[pos_h[0], pos_w[0]],  inputs[pos_h[1], pos_w[1]] and so on. PyTorch allows you to do that automatically by providing the indices as lists: inputs[pos_h, pos_w], where pos_h and pos_w have the same length. All you need to do is split your pos into two separate tensors, one for the indices along the height dimension and the other along the width dimension, which you also did in the TensorFlow version.

inputs = torch.randn(4, 4, 128)
# Random positions 0-3, shape: [12, 2]
pos = torch.randint(4, (12, 2))

# Positions split by dimension
pos_h = pos[:, 0]
pos_w = pos[:, 1]

# Index the inputs with the indices per dimension
gathered = inputs[pos_h, pos_w]

# Verify that it's identical to TensorFlow's output
inputs_tf = tf.convert_to_tensor(inputs.numpy())
pos_tf = tf.convert_to_tensor(pos.numpy())
gathered_tf = tf.gather_nd(inputs_tf, pos_tf)
gathered_tf = torch.from_numpy(gathered_tf.numpy())

torch.equal(gathered_tf, gathered) # => True

If you want to apply it to a tensor of size [C, H, W] instead, you only need to change the dimensions you want to index:

# For [H, W, C]
gathered = inputs[pos_h, pos_w]

# For [C, H, W]
gathered = inputs[:, pos_h, pos_w]

Batched version

Making it a batched batched version (for [N, H, W, C] or [N, C, H, W]) is not that difficult, and using that is more appropriate, since you're dealing with batches anyway. The only tricky part is that each element in the batch should only be applied to the corresponding batch. For this the batch dimensions needs to be enumerated, which can be done with torch.arange. The batch enumeration is just the list with the batch indices, which will be combined with the pos_h and pos_w indices, resulting in inputs[0, pos_h[0, 0], pos_h[0, 0]], inputs[0, pos_h[0, 1], pos_h[0, 1]] ... inputs[1, pos_h[1, 0], pos_h[1, 0]] etc.

batch_size = 3
inputs = torch.randn(batch_size, 4, 4, 128)
# Random positions 0-3, different for each batch, shape: [3, 12, 2]
pos = torch.randint(4, (batch_size, 12, 2))

# Positions split by dimension
pos_h = pos[:, :, 0]
pos_w = pos[:, :, 1]

batch_enumeration = torch.arange(batch_size) # => [0, 1, 2]
# pos_h and pos_w have shape [3, 12], so the batch enumeration needs to be
# repeated 12 times per batch.
# Unsqueeze to get shape [3, 1], now the 1 could be repeated to 12, but
# broadcasting will do that automatically.
batch_enumeration = batch_enumeration.unsqueeze(1)
# Index the inputs with the indices per dimension
gathered = inputs[batch_enumeration, pos_h, pos_w]

# Again, verify that it's identical to TensorFlow's output
inputs_tf = tf.convert_to_tensor(inputs.numpy())
pos_tf = tf.convert_to_tensor(pos.numpy())
# This time with batch_dims=1
gathered_tf = tf.gather_nd(inputs_tf, pos_tf, batch_dims=1)
gathered_tf = torch.from_numpy(gathered_tf.numpy())

torch.equal(gathered_tf, gathered) # => True

Again, for [N, C, H, W], only the dimensions that are indexed need to be changed:

# For [N, H, W, C]
gathered = inputs[batch_enumeration, pos_h, pos_w]

# For [N, C, H, W]
gathered = inputs[batch_enumeration, :, pos_h, pos_w]

Just a little side note on the interpolate implementation, rounding the positions (floor and ceil respectively) doesn't make sense, because indices must be integers, so it has no effect, as long as your positions are actual indices. That also results in i_top_left and i_bottom_left being the same value, but even if they are to be rounded differently, they are always 1 position apart. Furthermore, i_top_left and i_top_right are literally the same. I don't think that this function produces a meaningful output. I don't know what you're trying to achieve, but if you're looking for image interpolation you could have a look at torch.nn.functional.interpolate.


Instruction: How to convert a matrix of torch.tensor to a larger tensor?

Input:
I meet a problem to convert a python matrix of torch.tensor to a torch.tensor

For example, M is an (n,m) matrix, with each element M[i][j] is a torch.tensor with same size (p, q, r, ...). How to convert python list of list M to a torch.tensor with size (n,m,p,q,r,...)
e.g.

M = []
for i in range(5):
    row = []
    for j in range(10):
        row.append(torch.rand(3,4))
    M.append(row)

How to convert above M to a torch.tensor with size (5,10,3,4).

Output:
Try torch.stack() to stack a list of tensors on the first dimension.

import torch

M = []
for i in range(5):
    row = []
    for j in range(10):
        row.append(torch.rand(3,4))
    row = torch.stack(row)
    M.append(row)
M = torch.stack(M)

print(M.size())
# torch.Size([5, 10, 3, 4])


Instruction: Retrieving documents based on matrix multiplication

Input:
I have a model that represents a collection of documents in multidimensional vector space. So, for example, for 100k documents, my model represents them in the form of 300 dimensional vectors. So, finally, I get a matrix of size [100K, 300]. For retrieving those documents according to relevance to the given query,  I do matrix multiplication. For example, I represent a given query as a [300, 1]. Then I get the cosine similarity scores using matrix multiplication as follows :
[100K, 300]*[300, 1] = [100K, 1].
Now how can I retrieve top 1000 documents from this collection with highest cosine similarity. The trivial way would be to sort based on cosine similarity and grab the first 1000 docs. Is there any way to retrieve the documents this way using some function in pytorch?

I mean, how can I get the indices of highest 1000 values from a 1D torch tensor?p

Output:
Once you have the similarity scores after the dot product.
you can get the top 1000 indices as follows

top_indices = torch.argsort(sims)[:1000]
similar_docs = sims[top_indices]


Instruction: Pytorch Siamese Network not converging

Input:
Good morning everyone
Below is my implementation of a pytorch siamese network. I am using 32 batch size, MSE loss and SGD with 0.9 momentum as optimizer.
class SiameseCNN(nn.Module):
    def __init__(self):
        super(SiameseCNN, self).__init__()                                      # 1, 40, 50
        self.convnet = nn.Sequential(nn.Conv2d(1, 8, 7), nn.ReLU(),             # 8, 34, 44
                                    nn.Conv2d(8, 16, 5), nn.ReLU(),             # 16, 30, 40
                                    nn.MaxPool2d(2, 2),                         # 16, 15, 20
                                    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), # 32, 15, 20
                                    nn.Conv2d(32, 64, 3, padding=1), nn.ReLU()) # 64, 15, 20
        self.linear1 = nn.Sequential(nn.Linear(64 * 15 * 20, 100), nn.ReLU())
        self.linear2 = nn.Sequential(nn.Linear(100, 2), nn.ReLU())

    def forward(self, data):
        res = []
        for j in range(2):
            x = self.convnet(data[:, j, :, :])
            x = x.view(-1, 64 * 15 * 20)
            res.append(self.linear1(x))
        fres = abs(res[1] - res[0])
        return self.linear2(fres)

Each batch contains alternating pairs, i.e [pos, pos], [pos, neg], [pos, pos] etc... However, the network doesn't converge, and the problem seems that fres in the network is the same for each pair (regardless of whether it is a positive or negative pair), and the output of self.linear2(fres) is always approximately equal to [0.0531, 0.0770]. This is in contrast with what I am expecting, which is that the first value of [0.0531, 0.0770] would get closer to 1 for a positive pair as the network learns, and the second value would get closer to 1 for a negative pair. These two values also need to sum up to 1.
I have tested exactly the same setup and same input images for a 2 channel network architecture, where, instead of feeding in [pos, pos] you would stack those 2 images in a depth-wise fashion, for example numpy.stack([pos, pos], -1). The dimension of nn.Conv2d(1, 8, 7) also changes to nn.Conv2d(2, 8, 7) in this setup. This works perfectly fine.
I have also tested exactly the same setup and input images for a traditional CNN approach, where I just pass in single positive and negative grey scale images into the network, instead of stacking them (as with the 2-CH approach) or passing them in as image pairs (as with the Siamese approach). This also works perfectly, but the results are not so good as with the 2 channel approach.
EDIT (Solutions I've tried):

I have tried a number of different loss functions, including HingeEmbeddingLoss and CrossEntropyLoss, all resulting in more or less the same problem. So I think it is safe to say that the problem is not caused by the employed loss function; MSELoss.
Different batch sizes also seem to have no effect on the issue.
I tried increasing the number of trainable parameters as suggested in
Keras Model for Siamese Network not Learning and always predicting the same ouput
Also doesn't work.
Tried to change the network architecture as implemented here: https://github.com/benmyara/pytorch-examples/blob/master/notebooks/1_NeuralNetworks/9_siamese_nn.ipynb. In other words, changed the forward pass to the following code. Also changed the loss to CrossEntropy, and the optimizer to Adam. Still no luck:

def forward(self, data):
    res = []
    for j in range(2):
        x = self.convnet(data[:, j, :, :])
        x = x.view(-1, 64 * 15 * 20)
        res.append(x)
    fres = self.linear2(self.linear1(abs(res[1] - res[0]))))
    return fres

I also tried to change the whole network from a CNN to a linear network as implemented here: https://github.com/benmyara/pytorch-examples/blob/master/notebooks/1_NeuralNetworks/9_siamese_nn.ipynb. Still doesn't work.
Tried to use a lot more data as suggested here: Keras Model for Siamese Network not Learning and always predicting the same ouput. No luck...
Tried to use torch.nn.PairwiseDistance between the outputs of convnet. Made some sort of improvement; the network starts to converge for the first few epochs, and then hits the same plateau everytime:

def forward(self, data):
    res = []
    for j in range(2):
        x = self.convnet(data[:, j, :, :])
        res.append(x)
    pdist = nn.PairwiseDistance(p=2)
    diff = pdist(res[1], res[0])
    diff = diff.view(-1, 64 * 15 * 10)
    fres = self.linear2(self.linear1(diff))
    return fres

Another thing to note perhaps is that, within the context of my research, a Siamese network is trained for each object. So the first class is associated with the images containing the object in question, and the second class is associated with images containing other objects. Don't know if this might be the cause of the problem. It is however not a problem within the context of the Traditional CNN and 2-Channel CNN approaches.
As per request, here is my training code:
model = SiameseCNN().cuda()
ls_fn = torch.nn.BCELoss()
optim = torch.optim.SGD(model.parameters(),  lr=1e-6, momentum=0.9)
epochs = np.arange(100)
eloss = []
for epoch in epochs:
    model.train()
    train_loss = []
    for x_batch, y_batch in dp.train_set:
        x_var, y_var = Variable(x_batch.cuda()), Variable(y_batch.cuda())
        y_pred = model(x_var)
        loss = ls_fn(y_pred, y_var)
        train_loss.append(abs(loss.item()))
        optim.zero_grad()
        loss.backward()
        optim.step()
    eloss.append(np.mean(train_loss))
    print(epoch, np.mean(train_loss))

Note dp in dp.train_set is a class with attributes train_set, valid_set, test_set, where each set is created as follows:
DataLoader(TensorDataset(torch.Tensor(x), torch.Tensor(y)), batch_size=bs)

As per request, here is an example of the predicted probabilities vs true label, where you can see the model doesn't seem to be learning:
Predicted:  0.5030623078346252 Label:  1.0
Predicted:  0.5030624270439148 Label:  0.0
Predicted:  0.5030624270439148 Label:  1.0
Predicted:  0.5030625462532043 Label:  0.0
Predicted:  0.5030625462532043 Label:  1.0
Predicted:  0.5030626654624939 Label:  0.0
Predicted:  0.5030626058578491 Label:  1.0
Predicted:  0.5030627250671387 Label:  0.0
Predicted:  0.5030626654624939 Label:  1.0
Predicted:  0.5030627846717834 Label:  0.0
Predicted:  0.5030627250671387 Label:  1.0
Predicted:  0.5030627846717834 Label:  0.0
Predicted:  0.5030627250671387 Label:  1.0
Predicted:  0.5030628442764282 Label:  0.0
Predicted:  0.5030627846717834 Label:  1.0
Predicted:  0.5030628442764282 Label:  0.0

Output:
Problem solved. Turns out the network will predict the same output every time if you give it the same images every time  Small indexing mistake on my part during data partitioning. Thanks for everyone's help and assistance. Here is an example of the convergence as it is now:
0 0.20198837077617646
1 0.17636818194389342
2 0.15786472541093827
3 0.1412761415243149
4 0.126698794901371
5 0.11397973036766053
6 0.10332610329985618
7 0.09474560652673245
8 0.08779258838295936
9 0.08199785630404949
10 0.07704121413826942
11 0.07276330365240574
12 0.06907484836131335
13 0.06584368328005076
14 0.06295975042134523
15 0.06039590438082814
16 0.058096024941653016


Instruction: Runtime error, shape is invalid for input

Input:
i am trying to get the input and output information of a network. When debugging, i got this error, Runtime, shape ‘[-1, 400]’ is invalid for input of size 384. I tried different values, but can’t find the correct value. Is there a way to solve this issue? Thanks.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*5*5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

input_shape = (3, 21,21)
        dummy_input = torch.randn(6,*input_shape)
        graph = torch.jit._get_trace_graph(model,  args=dummy_input, _force_outplace=False, _return_inputs_states=False)

Error message:

RuntimeError: shape '[-1, 400]' is invalid for input of size 384

Output:
The shape of the tensor after the convolutional layers is [6,16,2,2]. So you cannot reshape it to 16*5*5 before feeding them to the linear layers. You should change your network to the one given below if you want to use the same filter sizes as the original in the convolutional layers.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*2*2, 120) # changed the size
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*2*2) # changed the size
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


Instruction: Weak optimizers in Pytorch

Input:
Consider a simple line fitting a * x + b = x, where a, b are the optimized parameters and x is the observed vector given by

import torch
X = torch.randn(1000,1,1)

One can immediately see that the exact solution is a=1, b=0 for any x and it can be found as easily as:

import numpy as np
np.polyfit(X.numpy().flatten(), X.numpy().flatten(), 1)

I am trying now to find this solution by means of gradient descent in PyTorch, where the mean square error is used as an optimization criterion.

import matplotlib.pyplot as plt
import numpy as np

import torch
import torch.nn as nn
from torch.optim import Adam, SGD, Adagrad, ASGD

X = torch.randn(1000,1,1) # Sample data

class SimpleNet(nn.Module): # Trivial neural network containing two weights
    def __init__(self):
        super(SimpleNet, self).__init__()
        self.f1 = nn.Linear(1,1)

    def forward(self, x):
        x = self.f1(x)
        return x

# Testing default setting of 3 basic optimizers

K = 500
net = SimpleNet()
optimizer = Adam(params=net.parameters())
Adam_losses = []
optimizer.zero_grad()   # zero the gradient buffers
for k in range(K):
    for b in range(1): # single batch
        loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
        loss.backward()
        optimizer.step()
        Adam_losses.append(float(loss.detach()))

net = SimpleNet()
optimizer = SGD(params=net.parameters(), lr=0.0001)
SGD_losses = []
optimizer.zero_grad()   # zero the gradient buffers
for k in range(K):
    for b in range(1): # single batch
        loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
        loss.backward()
        optimizer.step()
        SGD_losses.append(float(loss.detach()))

net = SimpleNet()
optimizer = Adagrad(params=net.parameters())
Adagrad_losses = []
optimizer.zero_grad()   # zero the gradient buffers
for k in range(K):
    for b in range(1): # single batch
        loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
        loss.backward()
        optimizer.step()
        Adagrad_losses.append(float(loss.detach()))

The training progress in terms of loss evolution can be shown as

What is surprising for me is a very slow convergence of the algorithms in default setting. I have thus 2 questions:

1) Is it possible to achieve an arbitrary small error (loss) purely by means of some Pytorch optimizer? Since the loss function is convex, it should be definitely possible, however, I am not able to figure out, how to achieve this using PyTorch. Note that the above 3 optimizers cannot do that - see the loss progress in log scale for 20000 iterations:

2) I am wondering how the optimizers can work well in complex examples, when they does not work well even in this extremely simple example. Or (and that is the second question) is it something wrong in their application above that I missed?

Output:
The place where you called zero_grad is wrong. During each epoch, gradient is added to the previous one and backpropagated. This makes the loss oscillate as it gets closer, but previous gradient throws it off of the solution again.

Code below will easily perform the task:

import torch

X = torch.randn(1000,1,1)

net = SimpleNet()
optimizer = Adam(params=net.parameters())
for epoch in range(EPOCHS):
    optimizer.zero_grad()  # zero the gradient buffers
    loss = torch.mean((net.forward(X) - X) ** 2)
    if loss < 1e-8:
        print(epoch, loss)
        break
    loss.backward()
    optimizer.step()

  1) Is it possible to achieve an arbitrary small error (loss) purely by
  means of some Pytorch optimizer?

Yeah, precision above is reached in around ~1500 epochs, you can go lower up to the machine (float in this case) precision

  2) I am wondering how the optimizers can work well in complex
  examples, when they does not work well even in this extremely simple
  example.

Currently, we don't have anything better (at least wide spread) for network optimization than first order methods. Those are used as it's much faster to calculate gradient than Hessians for higher order methods. And complex, non-convex functions may have a lot of minima which kinda fulfill the task we threw at it, there is no need for global minima per se (although they may under some conditions, see this paper).


Instruction: Deep Learning NLP: "Efficient" BERT-like Implementations?

Input:
I work in a legacy corporate setting where I only have 16 core 64GB VM to work with on an NLP project.  I have a multi-label NLP text classification problem where I would really like to utilize a deep representation learning model like BERT, RoBERTa, ALBERT, etc.

I have approximately 200,000 documents that need to be labeled and I have annotated set of about 2,000 to use as the ground truth for training/testing/fine tuning.  I also have a much larger volume of domain related documents to use for pre-training.  I will need to do the pre-training from scratch most likely, since this in a clinical domain.  I am also open to pre-trained models if they might have a chance working with just fine-tuning like Hugging Face, etc..

What models and their implementations that are PyTorch or Keras compatible would folks suggest as a starting point?  Or is this a computational non-starter with my existing compute resources?

Output:
If you want to use your current setup, it will have no problem running a transformer model. You can reduce memory use by reducing the batch size, but at the cost of slower runs.

Alternatively, test your algorithm on google Colab which is free. Then open a GCP account, google will provide $300 dollars of free credits. Use this to create a GPU cloud instance and then run your algorithm there.

You probably want to use Albert or Distilbert from HuggingFace Transformers. Albert and Distilbert are both compute and memory optimized. HuggingFace has lot's of excellent examples.

Rule of thumb you want to avoid Language Model training from scratch. If possible fine tune the language model or better yet skip it and go straight to the training the classifier. Also, HuggingFace and others have MedicalBert, ScienceBert, and other specialized pretrained models.


Instruction: Having trouble with input dimensions for Pytorch LSTM with torchtext

Input:
Problem

I'm trying to build a text classifier network using LSTM. The error I'm getting is:

RuntimeError: Expected hidden[0] size (4, 600, 256), got (4, 64, 256)

Details

The data is json and looks like this:

{"cat": "music", "desc": "I'm in love with the song's intro!", "sent": "h"}

I'm using torchtext to load the data.

from torchtext import data
from torchtext import datasets

TEXT = data.Field(fix_length = 600)
LABEL = data.Field(fix_length = 10)

BATCH_SIZE = 64

fields = {
    'cat': ('c', LABEL),
    'desc': ('d', TEXT),
    'sent': ('s', LABEL),
}

My LSTM looks like this

EMBEDDING_DIM = 64
HIDDEN_DIM = 256
N_LAYERS = 4

MyLSTM(
  (embedding): Embedding(11967, 64)
  (lstm): LSTM(64, 256, num_layers=4, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.3, inplace=False)
  (fc): Linear(in_features=256, out_features=8, bias=True)
  (sig): Sigmoid()
)

I end up with the following dimensions for the inputs and labels

batch = list(train_iterator)[0]
inputs, labels = batch
print(inputs.shape) # torch.Size([600, 64])
print(labels.shape) # torch.Size([100, 2, 64])

And my initialized hidden tensor looks like:

hidden # [torch.Size([4, 64, 256]), torch.Size([4, 64, 256])]

Question

I'm trying to understand what the dimensions at each step should be.
Should the hidden dimension be initialized to (4, 600, 256) or (4, 64, 256)?

Output:
The documentation of nn.LSTM - Inputs explains what the dimensions are:

  h_0 of shape (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for each element in the batch. If the LSTM is bidirectional, num_directions should be 2, else it should be 1.

Therefore, your hidden state should have size (4, 64, 256), so you did that correctly. On the other hand, you are not providing the correct size for the input.

  input of shape (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence. See torch.nn.utils.rnn.pack_padded_sequence() or torch.nn.utils.rnn.pack_sequence() for details.

While it says that the size of the input needs to be (seq_len, batch, input_size), you've set batch_first=True in your LSTM, which swaps batch and seq_len. Therefore your input should have size (batch_size, seq_len, input_size), but that is not the case as your input has seq_len first (600) and batch second (64), which is the default in torchtext because that's the more common representation, which also matches the default behaviour of LSTM.

You need to set batch_first=False in your LSTM.

Alternatively. if you prefer having batch as the first dimension in general, torch.data.Field also has the batch_first option.


Instruction: I am having hard time understanding the BCEWITHLOGITLOSS

Input: Hello everyone recently i am working on a assignment where i have to predict mask of image using foreground_background image,background image.I used bcewithlogitloss bcz i changed my target value as combination of 1 and 0 where 1 is for foreground and 0 is for background.So my results are pretty good but i am not still convinced why i used bcewithlogitloss.It is a reconstruction problem where i am using probabilty distribution(bcewithlogitloss).SO can anyone plz hepl me why i can use bcewithlogitloss in this example.Thank you

Output:
The task you're describing is semantic segmentation, where the model predicts a mask for the image.

In the mask, for every input pixel, there are 2 possible categories. Either 0 (black) for background, and 1 (white) for foreground.

So, technically, it's a classification problem with 2 classes.

We commonly use binary cross-entropy for binary classification problems (why), hence BCE is a good fit here. Finally, the withLogit part means, the sigmoid activation will be applied internally, your model won't need a separate activation layer (details).


Instruction: Split a multiple dimensional pytorch tensor into "n" smaller tensors

Input:
Let's say I have a 5D tensor which has this shape for example : (1, 3, 10, 40, 1). I want to split it into smaller equal tensors (if possible) according to a certain dimension with a step equal to 1 while preserving the other dimensions.

Let's say for example I want to split it according to the fourth dimension (=40) where each tensor will have a size equal to 10. So the first tensor_1 will have values from 0->9, tensor_2 will have values from 1->10 and so on.

The 39 tensors will have these shapes :

Shape of tensor_1 : (1, 3, 10, 10, 1)
Shape of tensor_2 : (1, 3, 10, 10, 1)
Shape of tensor_3 : (1, 3, 10, 10, 1)
...
Shape of tensor_39 : (1, 3, 10, 10, 1)

Here's what I have tried :

a = torch.randn(1, 3, 10, 40, 1)

chunk_dim = 10
a_split = torch.chunk(a, chunk_dim, dim=3)

This gives me 4 tensors. How can I edit this so I'll have 39 tensors with a step = 1 like I explained ?

Output:
This creates overlapping tensors which what I wanted :

torch.unfold(dimension, size, step)


Instruction: Train-Valid-Test split for custom dataset using PyTorch and TorchVision

Input:
I have some image data for a binary classification task and the images are organised into 2 folders as data/model_data/class-A and data/model_data/class-B.

There are a total of N images. I want to have a 70/20/10 split for train/val/test.
I am using PyTorch and Torchvision for the task. Here is the code I have so far.

from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils, datasets, models

data_transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

model_dataset = datasets.ImageFolder(root, transform=data_transform)
train_count = int(0.7 * total_count)
valid_count = int(0.2 * total_count)
test_count = total_count - train_count - valid_count
train_dataset, valid_dataset, test_dataset = torch.utils.data.random_split(model_dataset, (train_count, valid_count, test_count))
train_dataset_loader = torch.utils.data.DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER)
valid_dataset_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER)
test_dataset_loader  = torch.utils.data.DataLoader(test_dataset , batch_size=BATCH_SIZE, shuffle=False,num_workers=NUM_WORKER)
dataloaders = {'train': train_dataset_loader, 'val': valid_dataset_loader, 'test': test_dataset_loader}

I feel that this isn't the correct way to be doing this because of 2 reasons.

I am applying the same transform to all the splits. (This is not what I want to do, obviously! The solution for this is most probably the answer here.)
Usually people first separate the original data into test/train and then they
separate train  into train/val, whereas I am directly separating the
original data into train/val/test. (Is this correct?)

So, my question is, is what I am doing correct? (Probably not)
And if it is not correct, how do I go about writing the data loaders to achieve the required splits, so that I can apply separate transforms to each of train/test/val?

Output:
  Usually people first separate the original data into test/train and
  then they separate train into train/val, whereas I am directly
  separating the original data into train/val/test. (Is this correct?)

Yes, it's fully correct, readable and totally fine all in all

  I am applying the same transform to all the splits. (This is not what
  I want to do, obviously! The solution for this is most probably the
  answer here.)

Yes, that answer is a possibility but it's pointlessly verbose tbh. You can use third party tool torchdata, simply instalable with:

pip install torchdata

Documentation can be found here (also disclaimer: I'm the author).

It allows you to map your transformations to any torch.utils.data.Dataset easily (in this case to train). Your code would look like that (only two lines have to change, check the comments, also formatted your code to follow it easier):

import torch
import torchvision

import torchdata as td

data_transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.RandomResizedCrop(224),
        torchvision.transforms.RandomHorizontalFlip(),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(
            mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]
        ),
    ]
)

# Single change, makes an instance of torchdata.Dataset
# Works just like PyTorch's torch.utils.data.Dataset, but has
# additional capabilities like .map, cache etc., see project's description
model_dataset = td.datasets.WrapDataset(torchvision.datasets.ImageFolder(root))
# Also you shouldn't use transforms here but below
train_count = int(0.7 * total_count)
valid_count = int(0.2 * total_count)
test_count = total_count - train_count - valid_count
train_dataset, valid_dataset, test_dataset = torch.utils.data.random_split(
    model_dataset, (train_count, valid_count, test_count)
)

# Apply transformations here only for train dataset

train_dataset = train_dataset.map(data_transform)

# Rest of the code goes the same

train_dataset_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER
)
valid_dataset_loader = torch.utils.data.DataLoader(
    valid_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER
)
test_dataset_loader = torch.utils.data.DataLoader(
    test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=NUM_WORKER
)
dataloaders = {
    "train": train_dataset_loader,
    "val": valid_dataset_loader,
    "test": test_dataset_loader,
}

And yeah, I agree that specifying transform before splitting isn't too clear and IMO this is way more readable.


Instruction: Is it required to clear GPU tensors in PyTorch?

Input:
I am new to PyTorch, and I am exploring the functionality of .to() method. As per the documentation for the CUDA tensors, I see that it is possible to transfer the tensors between the CPU and GPU memory.

# let us run this cell only if CUDA is available
if torch.cuda.is_available():

    # creates a LongTensor and transfers it to GPU as torch.cuda.LongTensor
    a = torch.full((10,), 3, device=torch.device("cuda"))
    # transfers it to CPU, back to being a torch.LongTensor
    b = a.to(torch.device("cpu"))

In this context, I would like to know if it always necessary to transfer back the tensors from the GPU to CPU, perhaps to free the GPU memory? Doesn't, the runtime automatically clear the GPU memory?

Apart from its use of transferring data between CPU and GPU, I would like to know the recommended usage for .to() method (from the memory perspective). Thanks in advance.

Output:
  In this context, I would like to know if it always necessary to
  transfer back the tensors from the GPU to CPU, perhaps to free the GPU
  memory?

No, it's not always necessary. Memory should be freed when there are no more references to GPU tensor. Tensor should be cleared automatically in this case:

def foo():
    my_tensor = torch.tensor([1.2]).cuda()
    return "whatever"

smth = foo()

but it won't in this case:

def bar():
    return torch.tensor([1.2]).cuda()

tensor = bar()

In the second case (tensor being passed around, possibly accumulated or added to list), you should cast it to CPU in order not to waste GPU memory.

  Apart from its use of transferring data between CPU and GPU, I would
  like to know the recommended usage for .to() method (from the memory
  perspective)

Not sure what you mean here. What you should be after is the least to calls as they require copying the array (O(n) complexity), but shouldn't be too costly anyway (in comparison to pushing data through neural network for example) and probably not worth to be too hardcore about this micro-optimization.

Usually data loading is done on CPU (transformations, augmentations) and each batch copied to GPU (possibly with pinned memory) just before it is passed to neural network.

Also, as of 1.5.0 release, pytorch provides memory_format argument in .to method. This allows you to specify whether (N, C, H, W) (PyTorch default) or channel last (N, H, W, C) should be used for tensor and model (convolutional models with torch.nn.Conv2d to be precise). This could further speed up your models (for torchvision midels speedup of 16% was reported IIRC), see here for more info and usage.


Instruction: How to divide the dataset when it is distributed

Input:
Now I want to divide a dataset into two parts: the train set and validation set. I know that on a single GPU I can do this using a sampler:

indices = list(range(len(train_data)))
train_loader = torch.utils.data.DataLoader(
      train_data, batch_size=args.batch_size,
      sampler=torch.utils.data.sampler.SubsetRandomSampler(indices[:split]),
      pin_memory=True, num_workers=2)

But when I want to train it in a parallel way using torch.distributed, I have to use another sampler, namely, sampler = torch.utils.data.distributed.DistributedSampler(train_data)

So how should I do to use the two samplers, so that I can divide the dataset and distribute it at the same time?

Thank you very much for any help!

Output:
You can split torch.utils.data.Dataset before creating torch.utils.data.DataLoader.

Simply use torch.utils.data.random_split like this:

train, validation =
    torch.utils.data.random_split(
        dataset,
        (len(dataset)-val_length, val_length)
    )

This would give you two separate datasets which could be used with dataloaders however you wish.


Instruction: Passing weights to cross entropy loss

Input:
I am trying to assign different weights to different classes, so I have modified my loss criterion as such:

I had to convert the weight tensor to double torch.DoubleTensor(weight) since my model is already moved to double(). Am I doing it correctly ?

weights = [0.4,0.8,1.0]

class_weights = torch.DoubleTensor(weights).cuda()

criterion = nn.CrossEntropyLoss(weight=class_weights)

Output:
According to the documentation, the weight parameter to CrossEntropyLoss should be:

  weight (Tensor, optional) – a manual rescaling weight given to each class. If given, has to be a Tensor of size C.

I assume you have 3 classes (C=3). By the way, are you sure your model is moved to double()? Otherwise, you should prefer using FloatTensor. For example,

nn.CrossEntropyLoss(weight=torch.FloatTensor([0.4, 0.8, 1.0]))

Otherwise, your example looks ok!


Instruction: Can you train a BERT model from scratch with task specific architecture?

Input:
BERT pre-training of the base-model is done by a language modeling approach, where we mask certain percent of tokens in a sentence, and we make the model learn those missing mask. Then, I think in order to do downstream tasks, we add a newly initialized layer and we fine-tune the model.

However, suppose we have a gigantic dataset for sentence classification. Theoretically, can we initialize the BERT base architecture from scratch, train both the additional downstream task specific layer + the base model weights form scratch with this sentence classification dataset only, and still achieve a good result?

Thanks.

Output:
BERT can be viewed as a language encoder, which is trained on a humongous amount of data to learn the language well. As we know, the original BERT model was trained on the entire English Wikipedia and Book corpus, which sums to 3,300M words. BERT-base has 109M model parameters. So, if you think you have large enough data to train BERT, then the answer to your question is yes.

However, when you said "still achieve a good result", I assume you are comparing against the original BERT model. In that case, the answer lies in the size of the training data.

I am wondering why do you prefer to train BERT from scratch instead of fine-tuning it? Is it because you are afraid of the domain adaptation issue? If not, pre-trained BERT is perhaps a better starting point.

Please note, if you want to train BERT from scratch, you may consider a smaller architecture. You may find the following papers useful.

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations


Instruction: RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected input[1, 4, 160, 40] to have 3 channels, but got 4 channels instead

Input:
I am trying to use a pre-trained model. Here's where the problem occurs

Isn't the model supposed to take in a simple colored image? Why is it expecting a 4-dimensional input?

The eval() method is:

def eval(file):
    image = io.imread(file)
    plt.imshow(image)
    image = cv2.resize(image, (160,40)).transpose((2,1,0))
    output = model(torch.tensor(image[np.newaxis,...]).float())[0].squeeze().detach().numpy()
    return decode_prob(output)

And the output of eval('image.png') :

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-66-0b951c2596f8> in <module>()
----> 1 eval('/content/image.png')

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    344                             _pair(0), self.dilation, self.groups)
    345         return F.conv2d(input, weight, self.bias, self.stride,
--> 346                         self.padding, self.dilation, self.groups)
    347
    348     def forward(self, input):

RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected input[1, 4, 160, 40] to have 3 channels, but got 4 channels instead

Output:
The image you are loading is a 4 channel Image.
You can read it with cv2 and convert it to 3 channel RGB image as

def eval(file):
    image = cv2.imread(file)
    image = cv2.cvtColor(image, cv2.COLOR_BGRA2RGB)
    plt.imshow(image)
    image = cv2.resize(image, (160,40)).transpose((2,1,0))
    output = model(torch.tensor(image[np.newaxis,...]).float())[0].squeeze().detach().numpy()
    return decode_prob(output)


Instruction: Pytorch transformation on MNIST dataset

Input:
I currently have a project with Weak Supervision where I need to put a "masking" in front of a dataset. My issue right now is that I don't exactly know how to do it. Let me explain further with some code and images.

I am using the MNIST dataset that I have to edit in this way. As you can see a middle square is cut out. The code below is used to edit the MNIST using a for loop.

for i in range(int(image_size/2-5),int(image_size/2+3)):
   for j in range(int(image_size/2-5),int(image_size/2+3)):
      image[i][j] = 0

However, I am currently not sure how I should use this in a dataloader transform. The code for the dataloader and transform is shown here:

transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])

train_dataset = torchvision.datasets.MNIST(
    root="~/torch_datasets", train=True, transform=transform, download=True
)

test_dataset = torchvision.datasets.MNIST(
    root="~/torch_datasets", train=False, transform=transform, download=True
)

train_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=128, shuffle=True, num_workers=4, pin_memory=True
)

test_loader = torch.utils.data.DataLoader(
    test_dataset, batch_size=32, shuffle=False, num_workers=4
)

def imshow(img):
    #img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

dataiter = iter(train_loader)
images, labels = dataiter.next()
imshow(torchvision.utils.make_grid(images))

So is there a straightforward way to apply the transform to the full dataset in the torchvision.transforms.Compose?

Output:
You can define any custom transformation and as a function and use torchvision.transforms.Lambda in the transformation pipeline.

def erase_middle(image: torch.Tensor) -> torch.Tensor:
    for i in range(int(image_size/2-5),int(image_size/2+3)):
        for j in range(int(image_size/2-5),int(image_size/2+3)):
            image[:, i, j] = 0
    return image

transform = torchvision.transforms.Compose(
    [
        # First transform it to a tensor
        torchvision.transforms.ToTensor(),
        # Then erase the middle
        torchvision.transforms.Lambda(erase_middle),
    ]
)

erase_middle can be made more generic, such that it works for images with varying sizes and that aren't necessarily square.

def erase_middle(image: torch.Tensor) -> torch.Tensor:
    _, height, width = image.size()
    x_start = width // 2 - 5
    x_end = width // 2 + 3
    y_start = height // 2 - 5
    y_end = height // 2 + 3
    # Using slices achieves the same as the for loops
    image[:, y_start:y_end, x_start:x_end] = 0
    return image


Instruction: Access lower dimensional encoded data of autoencoder

Input:
Here is an autoencoder I’m working on from tutorial:https://debuggercafe.com/implementing-deep-autoencoder-in-pytorch/

I’m just learning about autoencoders and I’ve modified the source encode a custom small dataset which consists of:

[0,1,0,1,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0]

It seems to work ok, but I’m unsure how to access the lower dimensional embedding values of dimension 2 (set by parameter out_features).

I've added a methods to the Autoencoder class to return the embedding , is this the recommended method of accessing the embedding's ?

Code:

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import MNIST
from torchvision.utils import save_image
import warnings
import os

# import packages
import os
import torch
import torchvision
import torch.nn as nn
import torchvision.transforms as transforms
import torch.optim as optim
import matplotlib.pyplot as plt
import torch.nn.functional as F

from torchvision import datasets
from torch.utils.data import DataLoader
from torchvision.utils import save_image
import numpy as np

# utility functions
def get_device():
    if torch.cuda.is_available():
        device = 'cuda:0'
    else:
        device = 'cpu'
    return device

device = get_device()

features = torch.tensor(np.array([ [0,1,0,1,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0] ])).float()
tic_tac_toe_data_loader = torch.utils.data.DataLoader(features, batch_size=1, shuffle=True)

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.fc1 = nn.Linear(in_features=9, out_features=2)

    def forward(self, x):
        return F.sigmoid(self.fc1(x))

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()
        self.fc1 = nn.Linear(in_features=2, out_features=9)

    def forward(self, x):
        return F.sigmoid(self.fc1(x))

class Autoencoder(nn.Module):

    def __init__(self):
        super(Autoencoder, self).__init__()
        self.fc1 = Encoder()
        self.fc2 = Decoder()

    def forward(self, x):
        return self.fc2(self.fc1(x))

net = Autoencoder()
net.to(device)

NUM_EPOCHS = 50
LEARNING_RATE = 1e-3

criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

# image transformations
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

outputs = None

def train(net, trainloader, NUM_EPOCHS):
    train_loss = []
    for epoch in range(NUM_EPOCHS):
        running_loss = 0.0
        for data in trainloader:
            img = data
            img = img.to(device)
            img = img.view(img.size(0), -1)
#             print('img.shape' , img.shape)
            optimizer.zero_grad()

            outputs = net(img)
            loss = criterion(outputs, img)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

        loss = running_loss / len(trainloader)
        train_loss.append(loss)

    return train_loss

# train the network
train_loss = train(net, tic_tac_toe_data_loader, NUM_EPOCHS)

I can access the lower dimensional embedding using

print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))

But is this using the trained weight values for the embedding ? If I call Encoder multiple times with same values:

print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))
print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))

different results are returned:

tensor([0.5083, 0.5020], grad_fn=<SigmoidBackward>)
tensor([0.4929, 0.6940], grad_fn=<SigmoidBackward>)

Why is this the case ? Is an extra training step being invoked as a result of calling Encoder ?

Output:
By calling Encoder() you are basically creating a new instance of the encoder everytime and the weights are randomly initialized each time.

Generally, you make one instance of it and train it, save the weights, and infer on it.

Also, for PyTorch, you need not call .forward(), but call the instance directly. Forward is called by it implicitly, including other hook methods if any.

enc = Encoder()
input = torch.from_numpy(np.asarray([0,1,0,1,0,1,0,1,0]).float()
print(enc(input))
print(enc(input))

Training pass happens when you pass the Encode() instance to train function. Calling Encoder() only creates a new object.
Since each object has it's own weights, and the weights are initialized randomly (see xavier and kaiming initialization), you are different outputs. Moving to a single object, you still have to explicitly train it with the train function.


Instruction: training and evaluating an stacked auto-encoder model in pytorch

Input:
I am trying to train a model in pytorch.

input: 686-array
first layer: 64-array
second layer: 2-array
output: predition either 1 or 0

this is what I have so far:

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder_softmax = nn.Sequential(
            nn.Linear(686, 256),
            nn.ReLU(True),
            nn.Linear(256, 2),
            nn.Softmax()
        )

    def forward(self, x):
        x = self.encoder_softmax(x)
        return x

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

net = net.to(device)

iterations = 10
learning_rate = 0.98
criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(
    net.parameters(), lr=learning_rate, weight_decay=1e-5)

for epoch in range(iterations):
    loss = 0.0
    print("train_dl len: ", len(train_dl))

    # net.train()
    for i, data in enumerate(train_dl, 0):
        inputs, labels, vectorize = data

        labels = labels.long().to(device)
        inputs = inputs.float().to(device)
        optimizer.zero_grad()
        outputs = net(inputs)

        train_loss = criterion(outputs, labels)

        train_loss.backward()
        optimizer.step()

        loss += train_loss.item()

    loss = loss / len(train_dl)

but when I train the model, the loss is not going down. What am I doing wrong?

Output:
You're using nn.CrossEntropyLoss as the loss function, which applies log-softmax, but you also apply softmax in the model:

self.encoder_softmax = nn.Sequential(
    nn.Linear(686, 256),
    nn.ReLU(True),
    nn.Linear(256, 2),
    nn.Softmax() # <- needs to be removed
)

The output of your model should be the raw logits, without the nn.Softmax.

You should also lower the learning rate, because a learning rate of 0.98 is very high, which makes the training much less stable and you'll likely see the loss oscillate. Are more appropriate learning rate would be in the magnitude of 0.01 or 0.001.


Instruction: How can I resize Olivetti Dataset images 64x64 to 32x32 ?? I am getting error

Input:
batch_boyut = 2

train_loader = torch.utils.data.DataLoader(
    X_egitim, batch_size=batch_boyut)
val_loader = torch.utils.data.DataLoader(
    X_val, batch_size=batch_boyut)

class CNNModule(nn.Module):
    def __init__(self):
        super(CNNModule, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 40)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = CNNModule()
print(model)
model.forward(X_egitim)

I am getting this error.How can I solve? How can resize olivetti dataset images from 64x64 to 32x32 ?
I think I should change x.view but I don't know how I can

x = x.view(-1, 16 * 5 * 5)

RuntimeError: shape '[-1, 400]' is invalid for input of size 865280

Output:
I assume your input shape is 320 x 1 x 64 x 64.

I think you need to understand what is the output shape of convolution and max-pool operations. In your model, you have 2 CNN layers, followed by the max-pooling layer.

First CNN and max-pool layer:

x = self.pool(f.relu(self.conv1(x)))

Since conv1 has a 2d kernel of size 5 x 5, the output shape of conv1 will be (6 x 60 x 60). After passing through the max-pooling layer with kernel size 2 x 2, you will get 6 x 30 x 30 shaped tensor. You can easily compute the output shape of tensor as follows.

64 x 64 -> 60 x 60 -> 30 x 30

Where, 60 = (64 - cnn_kernel_size + 1) and, 30 = 60 // max_pool_kernel_size.

Similarly, after the second CNN and max-pool layer:

x = self.pool(f.relu(self.conv2(x)))

The shape of x will be: 320 x 16 x 13 x 13. Here, 13 = 26 // 2 and 26 = 30 - 5 + 1. Now, when you are trying to reshape x as x.view(-1, 16 * 5 * 5), it is throwing the error.

So, either you should revise the following layers which would take input of shape -1, 16 * 13 * 13, or you revise the CNN layers to finally get output tensor of shape -1, 16 * 5 * 5.

Note, if you have an input tensor of shape 32 x 32, you won't face the problem. Why?

Because: 32 -> (32 - 5 + 1) // 2 -> 14 -> (14 - 5 + 1) // 2 -> 5

So, if you want to convert size 64 x 64 to 32 x 32, you may consider performing downscaling using some operations, e.g., max-pooling 2d (with kernel_size 2 x 2), a linear transformation, etc.


Instruction: HDF5 dataloading very slow. Causes zero % volatility in GPUs

Input:
I am using a custom PyTorch dataclass to load instances from a H5 dataset I created. However, it appears to be incredibly slow when loading samples. I have followed several bits of advice on dealing with large HDF5 datasets, but I am wondering whether I am doing something that is obviously wrong. I am deploying my code on Linux if that makes a difference. I am running the code on 4 GPUs with nn.dataparallel in place for my model. As the dataloading is very slow, the GPU Volatility is at 0%.  Here is my dataclass loader:

import h5py
from torch.utils import data

class Features_Dataset(data.Dataset):
    def __init__(self, archive, phase):
        self.archive = archive
        self.phase = phase

    def __getitem__(self, index):
        with h5py.File(self.archive, 'r', libver='latest', swmr=True) as archive:
            datum = archive[str(self.phase) + '_all_arrays'][index]
            label = archive[str(self.phase) + '_labels'][index]
            path = archive[str(self.phase) +  '_img_paths'][index]
            return datum, label, path

    def __len__(self):
        with h5py.File(self.archive, 'r', libver='latest', swmr=True) as archive:
            datum = archive[str(self.phase) + '_all_arrays']
            return len(datum)

if __name__ == '__main__':
    train_dataset = Features_Dataset(archive= "featuresdata/train.hdf5", phase= 'train')
    trainloader = data.DataLoader(train_dataset, num_workers=8, batch_size=128)
    print(len(trainloader))
    for i, (data, label, path) in enumerate(trainloader):
        print(path)

Am I missing something obvious? Is there a better way of loading instances rapidly?

EDIT:

Here is the updated dataclass however I now get a picling error when trying to use multiprocessing.

import h5py
from torch.utils import data
import torch.multiprocessing as mp
mp.set_start_method('spawn')

class Features_Dataset(data.Dataset):
    def __init__(self, archive, phase):
        self.archive = h5py.File(archive, 'r')
        self.labels = self.archive[str(phase) + '_labels']
        self.data = self.archive[str(phase) + '_all_arrays']
        self.img_paths = self.archive[str(phase) + '_img_paths']

    def __getitem__(self, index):
        datum = self.data[index]
        label = self.labels[index]
        path = self.img_paths[index]
        return datum, label, path

    def __len__(self):
        return len(self.data)

    def close(self):
        self.archive.close()

if __name__ == '__main__':
    train_dataset = Features_Dataset(archive= "featuresdata/train.hdf5", phase= 'train')
    trainloader = data.DataLoader(train_dataset, num_workers=2, batch_size=4)
    print(len(trainloader))
    for i, (data, label, path) in enumerate(trainloader):
        print(path)

Output: Can you not open file once in the init and store the file handler? Currently when ever you called get item or len, you will be always opening the file on each call.


Instruction: How to decide which mode to use for 'kaiming_normal' initialization

Input:
I have read several codes that do layer initialization using nn.init.kaiming_normal_() of PyTorch. Some codes use the fan in mode which is the default. Of the many examples, one can be found here and shown below.

init.kaiming_normal(m.weight.data, a=0, mode='fan_in')

However, sometimes I see people using the fan out mode as seen here and shown below.

if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')

Can someone give me some guidelines or tips to help me decide which mode to select? Further I am working on image super resolutions and denoising tasks using PyTorch and which mode will be more beneficial.

Output:
According to documentation:

  Choosing 'fan_in' preserves the magnitude of the variance of the
  weights in the forward pass. Choosing 'fan_out' preserves the
  magnitudes in the backwards pass.

and according to Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015):

  We  note  that  it  is  sufficient  to  use  either  Eqn.(14)  or
  Eqn.(10)

where Eqn.(10) and Eqn.(14) are fan_in and fan_out appropriately. Furthermore:

  This means that if the initialization properly scales the backward
  signal, then this is also the case for the forward signal; and vice
  versa. For all models in this paper, both forms can make them converge

so all in all it doesn't matter much but it's more about what you are after. I assume that if you suspect your backward pass might be more "chaotic" (greater variance) it is worth changing the mode to fan_out. This might happen when the loss oscillates a lot (e.g. very easy examples followed by very hard ones).

Correct choice of nonlinearity is more important, where nonlinearity is the activation you are using after the layer you are initializaing currently. Current defaults set it to leaky_relu with a=0, which is effectively the same as relu. If you are using leaky_relu you should change a to it's slope.


Instruction: Could not find a version that satisfies the requirement torch~=1.4.0 (from syft) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)

Input:
I get the above error while installing syft package through Anaconda command. I followed the below link exactly as is,
https://medium.com/secure-and-private-ai-writing-challenge/installing-pysyft-package-ffa1ff0ad83c

Following commands were used:

conda create -n pysyft python=3
conda activate pysyft
pip install syft

Some of the links suggested to update the Pytorch version. I have 1.5.0+cpu version already and updating didnt help. I also tried using pip install syft without creating a conda environment, that didnt solve the problem either.
I went through the below stackoverflow link too which explains similar error, but it didnt help either:
Issues installing PyTorch 1.4 - "No matching distribution found for torch===1.4.0"

Any advice? Thanks in advance

Output:
Not all PyTorch versions are available on Python's package registry PyPI. For instance, the CPU only version or any Windows version is only available on PyTorch's custom registry. Selecting one of these versions on PyTorch - Get Started Locally will give you an installation command including the custom registry. Installing PySyft also installs PyTorch and the specific version you're getting, requires adding the custom registry:

pip install syft -f https://download.pytorch.org/whl/torch_stable.html

You might need to add --user if you don't have write access to the system wide package location.


Instruction: torch.optim returns "ValueError: can't optimize a non-leaf Tensor" for multidimensional tensor

Input:
I am trying to optimize the translations of the vertices of a scene with torch.optim.adam. It is a code piece from the redner tutorial series, which works fine with the initial setting. It tries to optimize a scene with shifting all the vertices by the same value called translation. Here is the original code:

vertices = []
for obj in base:
    vertices.append(obj.vertices.clone())

def model(translation):
    for obj, v in zip(base, vertices):
        obj.vertices = v + translation
    # Assemble the 3D scene.
    scene = pyredner.Scene(camera = camera, objects = objects)
    # Render the scene.
    img = pyredner.render_albedo(scene)
    return img

# Initial guess
# Set requires_grad=True since we want to optimize them later

translation = torch.tensor([10.0, -10.0, 10.0], device = pyredner.get_device(), requires_grad=True)

init = model(translation)
# Visualize the initial guess

t_optimizer = torch.optim.Adam([translation], lr=0.5)

I tried to modify the code such that it calculates an individual translation for each of the  vertices. For this I applied the following modifications to the code above, that makes the shape of the translation from torch.Size([3]) to torch.Size([43380, 3]):

# translation = torch.tensor([10.0, -10.0, 10.0], device = pyredner.get_device(), requires_grad=True)
translation = base[0].vertices.clone().detach().requires_grad_(True)
translation[:] = 10.0

This introduces the ValueError: can't optimize a non-leaf Tensor. Could you please help me work around the problem.

PS: I am sorry for the long text, I am very new to this subject, and I wanted to state the problem as comprehensive as possible.

Output:
Only leaf tensors can be optimised. A leaf tensor is a tensor that was created at the beginning of a graph, i.e. there is no operation tracked in the graph to produce it. In other words, when you apply any operation to a tensor with requires_grad=True it keeps track of these operations to do the back propagation later. You cannot give one of these intermediate results to the optimiser.

An example shows that more clearly:

weight = torch.randn((2, 2), requires_grad=True)
# => tensor([[ 1.5559,  0.4560],
#            [-1.4852, -0.8837]], requires_grad=True)

weight.is_leaf # => True

result = weight * 2
# => tensor([[ 3.1118,  0.9121],
#            [-2.9705, -1.7675]], grad_fn=<MulBackward0>)
# grad_fn defines how to do the back propagation (kept track of the multiplication)

result.is_leaf # => False

The result in this example, cannot be optimised, since it's not a leaf tensor. Similarly, in your case translation is not a leaf tensor because of the operation you perform after it was created:

translation[:] = 10.0
translation.is_leaf # => False

This has grad_fn=<CopySlices> therefore it's not a leaf and you cannot pass it to the optimiser. To avoid that, you would have to create a new tensor from it that is detached from the graph.

# Not setting requires_grad, so that the next operation is not tracked
translation = base[0].vertices.clone().detach()
translation[:] = 10.0
# Now setting requires_grad so it is tracked in the graph and can be optimised
translation = translation.requires_grad_(True)

What you're really doing here, is create a new tensor filled with the value 10.0 with the same size as the vertices tensor. This can be achieved much easier with torch.full_like

translation = torch.full_like(base[0],vertices, 10.0, requires_grad=True)


Instruction: Why must use DataParallel when testing？

Input:
Train on the GPU, num_gpus is set to 1：

device_ids = list(range(num_gpus))
model = NestedUNet(opt.num_channel, 2).to(device)
model = nn.DataParallel(model, device_ids=device_ids)

Test on the CPU：

model = NestedUNet_Purn2(opt.num_channel, 2).to(dev)
device_ids = list(range(num_gpus))
model = torch.nn.DataParallel(model, device_ids=device_ids)
model_old = torch.load(path, map_location=dev)
pretrained_dict = model_old.state_dict()
model_dict = model.state_dict()
pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
model_dict.update(pretrained_dict)
model.load_state_dict(model_dict)

This will get the correct result, but when I delete：

device_ids = list(range(num_gpus))
model = torch.nn.DataParallel(model, device_ids=device_ids)

the result is wrong.

Output:
nn.DataParallel wraps the model, where the actual model is assigned to the module attribute. That also means that the keys in the state dict have a module. prefix.

Let's look at a very simplified version with just one convolution to see the difference:

class NestedUNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)

model = NestedUNet()

model.state_dict().keys() # => odict_keys(['conv1.weight', 'conv1.bias'])

# Wrap the model in DataParallel
model_dp = nn.DataParallel(model, device_ids=range(num_gpus))

model_dp.state_dict().keys() # => odict_keys(['module.conv1.weight', 'module.conv1.bias'])

The state dict you saved with nn.DataParallel does not line up with the regular model's state. You are merging the current state dict with the loaded state dict, that means that the loaded state is ignored, because the model does not have any attributes that belong to the keys and instead you are left with the randomly initialised model.

To avoid making that mistake, you shouldn't merge the state dicts, but rather directly apply it to the model, in which case there will be an error if the keys don't match.

RuntimeError: Error(s) in loading state_dict for NestedUNet:
        Missing key(s) in state_dict: "conv1.weight", "conv1.bias".
        Unexpected key(s) in state_dict: "module.conv1.weight", "module.conv1.bias".

To make the state dict that you have saved compatible, you can strip off the module. prefix:

pretrained_dict = {key.replace("module.", ""): value for key, value in pretrained_dict.items()}
model.load_state_dict(pretrained_dict)

You can also avoid this issue in the future by unwrapping the model from nn.DataParallel before saving its state, i.e. saving model.module.state_dict(). So you can always load the model first with its state and then later decide to put it into nn.DataParallel if you wanted to use multiple GPUs.


Instruction: Convert from .npy to MLMultiArray for CoreML prediction in swift

Input: I have exported a PyTorch model to CoreML and want to do inference in swift. I have my input data stored on disk as a 2D float32 numpy ndarray .npy and need load into a MLMultiArray in swift. Is there a convenient way to do this?

Output:
Instead of saving as .npy (which is pickled), save the raw data from NumPy:

array.astype(np.float32).tofile(filename)

Now you can simply load this into a Data object in Swift and copy that into the MLMultiArray.


Instruction: NameError: name 'transforms' is not defined, while using google colab

Input:
train = datasets.MNIST("", train=True, download=True,transform = transforms.Compose([transforms.ToTensor()]))
test = datasets.MNIST("", train=False, download=True,transform = transforms.Compose([transforms.ToTensor()]))`

After executing this on colab notebook, I am getting this error:

Traceback (most recent call last)
<ipython-input-6-b81aa6cf1cbe> in <module>()
----> 1 train = datasets.MNIST("", train=True, download=True,transform = transforms.Compose([transforms.ToTensor()]))
      2 test = datasets.MNIST("", train=False, download=True,transform = transforms.Compose([transforms.ToTensor()]))

NameError: name 'transforms' is not defined*

Output:
I'm guessing from the context that you're using Pytorch, in which case you need to make sure you have:

from torchvision import transforms

In your imports. By the looks of things you have already imported datasets from the same library.


Instruction: PyTorch: one dimension in input data

Input:
I don't understand why this code works:

# Hyperparameters for our network
input_size = 784
hidden_sizes = [128, 64]
output_size = 10

# Build a feed-forward network
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))
print(model)

# Forward pass through the network and display output
images, labels = next(iter(trainloader))
images.resize_(images.shape[0], 1, 784)
ps = model.forward(images[0,:])

The size of an image is (images.shape[0], 1, 784) but our networks has input_size = 784. How does the network handle 1 dimension in an input image? I tried to change images.resize_(images.shape[0], 1, 784) to images = images.view(images.shape[0], -1) but I got an error:

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

For reference, data loader is created the next way:

# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5,), (0.5,)),
                              ])
# Download and load the training data
trainset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

Output:
PyTorch networks take inputs as [batch_size, input_dimensions]

In your case, images[0,:] is of the shape [1,784] where “1” is tue batch size and that is why your code works.


Instruction: pytorch object too deep for array when saving image

Input:
I am trying to run the code from the following github rep:

https://github.com/iamkrut/image_inpainting_resnet_unet

I havent changed anything in the code and it is causing a ValueError, that the object is too deep, when the code tries to save the image. The error seems to come from these two lines.

images = img_tensor.cpu().detach().permute(0,2,3,1)
plt.imsave(join(data_dir, 'samples', image), images[index,:,:,:3])

Here is the error statement

  File "train.py", line 205, in <module>
    data_dir=args.data_dir)
  File "train.py", line 94, in train_net
    plt.imsave(join(data_dir, 'samples', image), images[index,:,:,:]);
  File "C:\ProgramData\Anaconda3\envs\torch2\lib\site-packages\matplotlib\pyplot.py", line 2140, in imsave
    return matplotlib.image.imsave(fname, arr, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\torch2\lib\site-packages\matplotlib\image.py", line 1498, in imsave
    _png.write_png(rgba, fname, dpi=dpi)
ValueError: object too deep for desired array

Anyone know what could be causing this or how to fix it?
Thank you

Output:
matplotlib package does not understand the pytorch datatype (tensor). you should convert tensor array to numpy array and then use matplotlib functions.

a = torch.rand(10, 3, 20, 20)
plt.imsave("test.jpg", a.cpu().detach().permute(0, 2, 3, 1)[0, ...]) # Error
plt.imsave("test.jpg", a.cpu().detach().permute(0, 2, 3, 1).numpy()[0, ...])


Instruction: RuntimeError: shape '[1, 1024]' is invalid for input of size 50176

Input:
I am trying to use Alexnet over the CIFAR-10 dataset. I have resized my image to 224x224 which I'm guessing is the issue. Nevertheless, I get the following error:

<ipython-input-11-34884668038d> in forward(self, x)
     37     def forward(self, x):
     38         x = self.features(x)
---> 39         x = x.view(x.size(0), 256 * 2 * 2)
     40         x = self.classifier(x)
     41         return x

RuntimeError: shape '[1, 1024]' is invalid for input of size 50176

My Alexnet model code is as follows:

NUM_CLASSES = 10

class AlexNet(nn.Module):
    def __init__(self, num_classes=NUM_CLASSES):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, stride=2, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
            nn.Conv2d(64, 192, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
        )
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 2 * 2, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), 256 * 2 * 2)
        x = self.classifier(x)
        return x

Any help would be appreciated :)

Output: CIFAR nets expects input data much smaller than 224x224, usually 32x32.