Instruction: How to change activation layer in Pytorch pretrained module?

Input:
How to change the activation layer of a Pytorch pretrained network?
Here is my code :

print("All modules")
for child in net.children():
    if isinstance(child,nn.ReLU) or isinstance(child,nn.SELU):
        print(child)

print('Before changing activation')
for child in net.children():
    if isinstance(child,nn.ReLU) or isinstance(child,nn.SELU):
        print(child)
        child=nn.SELU()
        print(child)
print('after changing activation')
for child in net.children():
    if isinstance(child,nn.ReLU) or isinstance(child,nn.SELU):
        print(child)

Here is my output:

All modules
ReLU(inplace=True)
Before changing activation
ReLU(inplace=True)
SELU()
after changing activation
ReLU(inplace=True)

Output:
._modules solves the problem for me.

for name,child in net.named_children():
    if isinstance(child,nn.ReLU) or isinstance(child,nn.SELU):
        net._modules['relu'] = nn.SELU()


Instruction: Command line python and jupyter notebooks use two different versions of torch

Input:
On my conda environment importing torch from command line Python and from a jupyter notebook yields two different results.

Command line Python:

$ source activate GNN
(GNN) $ python
>>> import torch
>>> print(torch.__file__)
/home/riccardo/.local/lib/python3.7/site-packages/torch/__init__.py
>>> print(torch.__version__)
0.4.1

Jupyter:

(GNN) $ jupyter notebook --no-browser --port=8890

import torch

print(torch.__file__)
/home/riccardo/.local/lib/python3.6/site-packages/torch/__init__.py

print(torch.__version__)
1.2.0+cu92

I tried the steps suggested in Conda environments not showing up in Jupyter Notebook

$ conda install ipykernel
$ source activate GNN
(GNN) $ python -m ipykernel install --user --name GNN --display-name "Python (GNN)"
Installed kernelspec GNN in /home/riccardo/.local/share/jupyter/kernels/gnn

but that did not solve the problem.

Output:
You need to sort of make the Anaconda environment recognized in Jupyter using

conda activate myenv
conda install -n myenv ipykernel
python -m ipykernel install --user --name myenv --display-name "Python (myenv)"

Replace myenv with the name of your environment. Later on, in your Jupyter Notebook, in the Select Kernel option, you will see this Python (myenv) option.


Instruction: Is there really no padding=same option for PyTorch's Conv2d?

Input:
I'm currently working on building a convolutional neural network (CNN) that will work on financial time series data. The input shape is (100, 40) - 100 time stamps by 40 features.
The CNN that I'm using uses asymmetric kernel sizes (i.e. 1 x 2 and 4 x 1) and also asymmetric strides (i.e. 1 x 2 for the 1 x 2 layers and 1 x 1 for the 4 x 1 layers).
In order to maintain the height dimension to stay 100, I needed to pad the data. In my research, I noticed that people who use TensorFlow or Keras simply use padding='same'; but this option is apparently unavailable in PyTorch.
According to some answers in What is the difference between 'SAME' and 'VALID' padding in tf.nn.max_pool of tensorflow?, and also this answer on the PyTorch discussion forum, I can manually calculate how I need to pad my data, and use torch.nn.ZeroPad2d to solve the problem - since apparently normal torch.nn.Conv2d layers don't support asymmetric padding (I believe that the total padding I need is 3 in height and 0 in width).
I tried this code:
import torch
import torch.nn as nn

conv = nn.Conv2d(1, 1, kernel_size=(4, 1))
pad = nn.ZeroPad2d((0, 0, 2, 1)) # Add 2 to top and 1 to bottom.

x = torch.randint(low=0, high=9, size=(100, 40))
x = x.unsqueeze(0).unsqueeze(0)

y = pad(x)

x.shape # (1, 1, 100, 40)
y.shape # (1, 1, 103, 40)

print(conv(x.float()).shape)
print(conv(y.float()).shape)

# Output
# x -> (1, 1, 97, 40)
# y -> (1, 1, 100, 40)

It does work, in the sense that the data shape remains the same. However, is there really no padding='same' option available? Also, how can we decide which side to pad?

Output:
I had the same issue some time ago, so I implemented it myself using a ZeroPad2d layer as you are trying to do. Here is the right formula:
from functools import reduce
from operator import __add__

kernel_sizes = (4, 1)

# Internal parameters used to reproduce Tensorflow "Same" padding.
# For some reasons, padding dimensions are reversed wrt kernel sizes,
# first comes width then height in the 2D case.
conv_padding = reduce(__add__,
    [(k // 2 + (k - 2 * (k // 2)) - 1, k // 2) for k in kernel_sizes[::-1]])

pad = nn.ZeroPad2d(conv_padding)
conv = nn.Conv2d(1, 1, kernel_size=kernel_sizes)

print(x.shape) # (1, 1, 103, 40)
print(conv(y.float()).shape) # (1, 1, 103, 40)

Also, as mentioned by @akshayk07 and @Separius, I can confirm that it is the dynamic nature of pytorch that makes it hard. Here is a post about this point from a Pytorch developper.


Instruction: BERT sentence embedding by summing last 4 layers

Input:
I used Chris Mccormick tutorial on BERT using pytorch-pretained-bert to get a sentence embedding as follows:

tokenized_text = tokenizer.tokenize(marked_text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [1] * len(tokenized_text)
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
model = BertModel.from_pretrained('bert-base-uncased')
model.eval()

with torch.no_grad():
    encoded_layers, _ = model(tokens_tensor, segments_tensors)
    # Holds the list of 12 layer embeddings for each token
    # Will have the shape: [# tokens, # layers, # features]
    token_embeddings = []

    # For each token in the sentence...
    for token_i in range(len(tokenized_text)):
        # Holds 12 layers of hidden states for each token
        hidden_layers = []

        # For each of the 12 layers...
        for layer_i in range(len(encoded_layers)):

                # Lookup the vector for `token_i` in `layer_i`
                vec = encoded_layers[layer_i][batch_i][token_i]

                hidden_layers.append(vec)

        token_embeddings.append(hidden_layers)

Now, I am trying to get the final sentence embedding by summing the last 4 layers as follows:

summed_last_4_layers = [torch.sum(torch.stack(layer)[-4:], 0) for layer in token_embeddings]

But instead of getting a single torch vector of length 768 I get the following:

[tensor([-3.8930e+00, -3.2564e+00, -3.0373e-01,  2.6618e+00,  5.7803e-01,
-1.0007e+00, -2.3180e+00,  1.4215e+00,  2.6551e-01, -1.8784e+00,
-1.5268e+00,  3.6681e+00, ...., 3.9084e+00]), tensor([-2.0884e+00, -3.6244e-01,  ....2.5715e+00]), tensor([ 1.0816e+00,...-4.7801e+00]), tensor([ 1.2713e+00,.... 1.0275e+00]), tensor([-6.6105e+00,..., -2.9349e-01])]

What did I get here? How do I pool the sum of the last for layers?

Thank you!

Output:
You create a list using a list comprehension that iterates over token_embeddings. It is a list that contains one tensor per token - not one tensor per layer as you probably thought (judging from your for layer in token_embeddings). You thus get a list with a length equal to the number of tokens. For each token, you have a vector that is a sum of BERT embeddings from the last 4 layers.
More efficient would be avoiding the explicit for loops and list comprehenions:
summed_last_4_layers = torch.stack(encoded_layers[-4:]).sum(0)

Now, variable summed_last_4_layers contains the same data, but in the form of a single tensor of dimension: length of the sentence × 768.
To get a single (i.e., pooled) vector, you can do pooling over the first dimension of the tensor. Max-pooling or average-pooling might make much more sense in this case than summing all the token embeddings. When summing the values, vectors of differently long sentences are in different ranges and are not really comparable.


Instruction: Pytorch RuntimeError: size mismatch

Input:
I get the following error when I try to run this code for training a GAN for prediction:

  RuntimeError: size mismatch, m1: [128 x 1], m2: [1392 x 2784] at C:\w\1\s\tmp_conda_3.7_021303\conda\conda-bld\pytorch_1565316900252\work\aten\src\TH/generic/THTensorMath.cpp:752

Please comment if you see any other mistakes in the code or if you have some tips in general.

# Sample indices
def sample_idx(m, n):
    A = np.random.permutation(m)
    idx = A[:n]
    return idx

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.fc1 = nn.Linear(in_features = dim, out_features = dim*2)
        self.fc2 = nn.Linear(in_features = dim*2, out_features = dim)
        self.fc3 = nn.Linear(in_features = dim, out_features = int(dim/2))
        self.fc4 = nn.Linear(in_features = int(dim/2), out_features = 1)
        self.relu = nn.LeakyReLU(0.2)
        self.sigmoid = nn.Sigmoid()
        self.init_weight()

    def init_weight(self):
        layers = [self.fc1, self.fc2, self.fc3]
        [nn.init.xavier_normal_(layer.weight) for layer in layers]

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.relu(self.fc3(x))
        x = self.sigmoid(self.fc4(x))
        return x

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.fc1 = nn.Linear(in_features = dim, out_features = dim*2)
        self.fc2 = nn.Linear(in_features = dim*2, out_features = dim)
        self.fc3 = nn.Linear(in_features = dim, out_features = int(dim/2))
        self.fc4 = nn.Linear(in_features = int(dim/2), out_features = 1)
        self.relu = nn.LeakyReLU(0.2)
        self.init_weight()

    def init_weight(self):
        layers = [self.fc1, self.fc2, self.fc3]
        [nn.init.xavier_normal_(layer.weight) for layer in layers]

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.relu(self.fc3(x))
        x = self.fc4(x)
        return x

# 1. Mini batch size
mb_size = 128

# 4. Loss Hyperparameters
alpha = 10

# 5. Input Dim (Fixed)
dim = data.shape[1] - 1

# 6. Train Test Split
train_n = int(199476 * 0.8)
test_n = 199476 - int(199476 * 0.8)

X_train = data[:train_n,:-1]
y_train = data[:train_n, -1]

X_test = data[train_n:,:-1]
y_test = data[train_n:, -1]

# Nets
D = Discriminator()
G = Generator()

# Optimizers
optimD = torch.optim.Adam(D.parameters(), lr = 0.001)
optimG = torch.optim.Adam(G.parameters(), lr = 0.001)

# Loss
mse_loss = nn.MSELoss(reduction = 'elementwise_mean')

i = 1
for it in tqdm(range(10000)):
    mb_idx = sample_idx(train_n, mb_size)
    X_mb = X_train[mb_idx,:]
    X_mb = torch.tensor(X_mb).float()
    y_mb = y_train[mb_idx]
    y_mb = torch.tensor(y_mb).float()

    # Train D
    G_sample = G(X_mb)
    D_loss1 = ((D(X_mb) + 1e-8).log()).mean() + ((1 - D(G_sample) + 1e-8).log()).mean()
    D_loss = - D_loss1
    D_loss.backward()
    optimD.step()
    optimD.zero_grad()

    # Train G
    G_sample = G(X_mb)
    D_prob.detach()
    G_loss1 = ((1 - D(G_sample) + 1e-8).log()).mean()
    G_mse_loss = mse_loss(G_sample, y_mb)
    G_loss = G_loss1 + alpha * G_mse_loss
    G_loss.backward()
    optimG.step()
    optimG.zero_grad()

    if it % 100 == 0:
        print("Iter: {}".format(it))
        print("D_loss: {:.4}".format(D_loss))
        print("Train loss: {:.4}".format(G_mse_loss))
        print()

Output:
As you mentioned, you are getting the error in the following line.

D_loss1 = ((D(X_mb) + 1e-8).log()).mean() + ((1 - D(G_sample) + 1e-8).log()).mean()

  I doubt the problematic part is: D(G_sample). Why?

Because G_sample = G(X_mb) is of shape [batch_size, 1] which cannot be given as an input to the Discriminator, D because it takes tensor of shape [batch_size, dim] as input.

That's why you are getting the error:

RuntimeError: size mismatch, m1: [128 x 1], m2: [1392 x 2784]

As you can see, you have an input of shape, [128 x 1] where batch_size = 128. But the Discriminator D is expecting an input of shape [batch_size x 1392]. Here, m2 is the shape of the weight matrix of fc1 layer in the Discriminator.


Instruction: Any numpy/torch style to set value given an index ndarray and a flag ndarray?

Input:
I'm working on PyTorch and currently I met a problem for which I've no idea how to solve it in a torch/numpy style. For example, suppose I have three PyTorch tensors

import torch
import numpy as np

indices = torch.from_numpy(np.array([[2, 1, 3, 0], [1, 0, 3, 2]]))
flags = torch.from_numpy(np.array([[False, False, False, True], [False, False, True, True]]))
tensor = torch.from_numpy(np.array([[2.8, 0.5, 1.2, 0.9], [3.1, 2.8, 1.3, 2.5]]))

Here flags is a boolean flag tensor to show which elements in indices should be extracted. Given the extracted indices, I want to set the corresponding elements in tensor to an indicated const (say 1e-30). Based on the example shown above, I want

>>> sub_indices = indices.op1(flags)
>>> sub_indices
tensor([[0], [3, 2]])
>>> tensor.op2(sub_indices, 1e-30)
>>> tensor
tensor([[1e-30, 0.5, 1.2, 0.9], [3.1, 2.8, 1e-30, 1e-30]])

Could anyone help to give a solution? I'm using list comprehension but I think this way is a little bit ugly. I tried indices[flags] but it only returns a 1d-array [0, 3, 2] so applying this would change all rows on the same columns 0, 2, 3

Some additional remarks:

The number of "True" values for each row in flags cannot be determined
Each row of indices is assured to be a permutation of sequence 0 ... N - 1

Below is a numpy version of the example code, for the convenience of copy-pasting. I doubt whether this could be done in a pure numpy way

import numpy as np

indices = np.array([[2, 1, 3, 0], [1, 0, 3, 2]])
flags = np.array([[False, False, False, True], [False, False, True, True]])
tensor = np.array([[2.8, 0.5, 1.2, 0.9], [3.1, 2.8, 1.3, 2.5]])

Output:
You may sort flags according to the indices to create a mask, then use the mask as a mux. Here is an example code:

indices = np.array([[2, 1, 3, 0], [1, 0, 3, 2]])
flags = np.array([[False, False, False, True], [False, False, True, True]])
tensor = np.array([[2.8, 0.5, 1.2, 0.9], [3.1, 2.8, 1.3, 2.5]])

indices_sorted = indices.argsort(axis=1)
mask = np.take_along_axis(flags, indices_sorted, axis=1)
result = tensor * (1 - mask) + 1e-30 * mask

I'm not quite familiar with pytorch, but I guess it is not a good idea to gather a ragged tensor. Though, even in the worst case, you can convert to/from numpy arrays.


Instruction: NN regression loss value not decreasing

Input:
I'm training a NN with Pytorch to predict the expected price for the Boston dataset.
The network looks like this:

from sklearn.datasets import load_boston
from torch.utils.data.dataset import Dataset
from torch.utils.data import DataLoader
import torch.nn.functional as F
import torch
import torch.nn as nn
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(13, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, 32)
        self.fc4 = nn.Linear(32, 16)
        self.fc5 = nn.Linear(16,1)

    def forward(self, x):
       x = self.fc1(x)
       x = self.fc2(x)
       x = F.relu(x)
       x = self.fc3(x)
       x = F.relu(x)
       x = self.fc4(x)
       x = F.relu(x)
       return self.fc5(x)

And the data loader:

class BostonData(Dataset):
    __xs = []
    __ys = []

    def __init__(self, train = True):
        df = load_boston()
        index = int(len(df["data"]) * 0.7)
        if train:
            self.__xs = df["data"][0:index]
            self.__ys = df["target"][0:index]
        else:
            self.__xs = df["data"][index:]
            self.__ys = df["target"][index:]

    def __getitem__(self, index):
        return self.__xs[index], self.__ys[index]

    def __len__(self):
        return len(self.__xs)

In my first attempt I didn't add the ReLU units, but after a little bit of research I saw that adding them is a common practice, but It didn't work out for me.

Here is the training code:

dset_train = BostonData(train = True)
dset_test = BostonData(train = False)
train_loader = DataLoader(dset_train, batch_size=30, shuffle=True)
test_loader = DataLoader(dset_train, batch_size=30, shuffle=True)

optimizer = optim.Adam(net.parameters(), lr = 0.001)
criterion = torch.nn.MSELoss()
EPOCHS = 10000

lloss = []

for epoch in range(EPOCHS):
    for trainbatch in train_loader:
        X,y = trainbatch
        net.zero_grad()
        output = net(X.float())
        loss = criterion(output, y)
        loss.backward()
        optimizer.step()
    lloss.append(loss)
    print(loss)

After 10k epochs, the loss graph looks like the following

where I don't see any clear decrease.
I don't know if I'm messing up with the torch.nn.MSELoss(), the optimizer or maybe with the net topology, so any help will be appreciated.

Edit:
Changing the learning rate and normalizing the data didn't work for me. I added the line self.__xs = (self.__xs - self.__xs.mean()) / self.__xs.std()
 and a change to lr = 0.01. The loss plot is very similar to the first one.

Same plot for lr = 0.01 and normalizing after 1000 epochs:

Output:
you are appending lloss once per epoch and that correct but you are appending with loss (using only last batch) where you should append with avg_train_loss

Try:

for epoch in range(EPOCHS):
    avg_train_loss = 0
    for trainbatch in train_loader:
        X,y = trainbatch
        net.zero_grad()
        output = net(X.float())
        loss = criterion(output, y)
        loss.backward()
        optimizer.step()
        avg_train_loss += loss.item() / len(train_loader)
    lloss.append(avg_train_loss)


Instruction: Convert string to byte for pytorch loader

Input:
The method of downloading a pytorch model path is not in my control and I am trying to figure out a way to convert downloaded string data to byte data.  The code below downloads my saved model from Dropbox and uses bytes with utf-8 encoding to encode the string. The problem is when I use torch.load with BytesIO I get a UnpicklingError with invalid load key, '<'.

    data = bytes(self.Download("https://www.dropbox.com/s/exampleurl/checkpoint.pth?dl=1"), 'utf-8')

    self.agent.local.load_state_dict(torch.load(BytesIO(data ), map_location=lambda storage, loc: storage))

The code below worked perfectly until requests was disabled and I am now trying to use the method above.

    dropbox_url = "https://www.dropbox.com/s/exampleurl/checkpoint.pth?dl=1"

    data = requests.get(dropbox_url )

    self.agent.local.load_state_dict(torch.load(BytesIO(data.content), map_location=lambda storage, loc: storage))

I just need to figure out a way to convert the string to bytes data the correct way.

Output:
I had to convert the byte data to base64 and save the file in that format.  Once I uploaded on Dropbox and downloaded using the built in method, I converted the base64 file back to bytes and it worked!

import base64
from io import BytesIO

with open("checkpoint.pth", "rb") as f:
    byte = f.read(1)

# Base64 Encode the bytes
data_e = base64.b64encode(byte)

filename ='base64_checkpoint.pth'

with open(filename, "wb") as output:
    output.write(data_e)

# Save file to Dropbox

# Download file on server
b64_str= self.Download('url')

# String Encode to bytes
byte_data = b64_str.encode("UTF-8")

# Decoding the Base64 bytes
str_decoded = base64.b64decode(byte_data)

# String Encode to bytes
byte_decoded = str_decoded.encode("UTF-8")

# Decoding the Base64 bytes
decoded = base64.b64decode(byte_decoded)

torch.load(BytesIO(decoded))


Instruction: Unable to download saved model from online resource, pickle error

Input:
I am unable to download and use a model I saved earlier from a online-repository. Here's the code:

model = Model().double()   # Model is defined in another class
state_dict = torch.hub.load_state_dict_from_url(r'https://filebin.net/j2977ux7kts41aft/checkpoint_best.pt?t=wjbujfoo')
model.load_state_dict(state_dict)
model.eval()

Which gives me the following error:

Traceback (most recent call last):
  File "/path/file.py", line 47, in <module>
    state_dict = torch.hub.load_state_dict_from_url(r'https://filebin.net/j2977ux7kts41aft/checkpoint_best.pt?t=wjbujfoo')
  File "anaconda3/envs/torch_env/lib/python3.6/site-packages/torch/hub.py", line 466, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location)
  File "/anaconda3/envs/torch_env/lib/python3.6/site-packages/torch/serialization.py", line 386, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "anaconda3/envs/torch_env/lib/python3.6/site-packages/torch/serialization.py", line 563, in _load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\x0a'.

The model resides in:
https://filebin.net/j2977ux7kts41aft/checkpoint_best.pt?t=wjbujfoo

Note that I can perfectly download it manually, and then use torch.load(path) to load it without errors, but I need to do it from code! Could it be that the serializing when downloading from url somehow messes up the pickle encoding?

Edit: I don't have to use filebin, any online-storage that supports what I try to do will suffice.

Output:
The problem was indeed within the environment configuration. I created the model with PyTorch 1.0.2 and then updated to 1.2.0 in order to use torch.hub. This gave me the pickle error. After training a new model in 1.2.0, the error is now gone.

Hope this help someone in the future :)


Instruction: How to transform network's output images to colored pictures with target is PIL's P mode?

Input:
The original value in target images is 20 and 16 at training process with P mode of PIL,so I transform 20 to 1 and 16 to 2 in order to train the segmentation task.
But when I want to get the output images,the pictures aren't colored although I used the code

        pred=pred.reshape([512,512]).astype('uint8')
        (x, y) = pred.shape
        for xx in range(x):
            for yy in range(y):
                if pred[xx, yy] == 2:
                    pred[xx, yy] = 16
                elif pred[xx, yy] == 1:
                    pred[xx, yy] = 20
        pp = Image.fromarray(pred).convert('P')
        pp.save(r'E:\python_workspace\0711\run\pascal\{}.png'.format(i))

But the output image is

I have see the value with PIL.open and transform it to numpy to see the values,the part of things is convert to 16 and 20,the mode is P too.
How can I deal with this problem?

Output:
You appear to have managed to change all pixels with index 20 into index 1 and all pixels with index 16 into 2. However, you then need to copy the palette entry 20 to palette entry 1 and palette entry 16 to palette entry 2 in order to make the colours remain the same.

So, you want:

import numpy as np
from PIL import Image

# Load image
im = Image.open('R0T9R.png')

# Get palette and make into Numpy array of 256 entries of 3 RGB colours
palette = np.array(im.getpalette(),dtype=np.uint8).reshape((256,3))

# Set palette entry 1 the same as entry 20, and 2 the same as 16
palette[1] = palette[20]
palette[2] = palette[16]

# Change pixels too - this replaces your slow "for" loops
npim = np.array(im)
npim[npim==16] = 2
npim[npim==20] = 1

# Make Numpy array back into image
res = Image.fromarray(npim)

# Apply our modified palette and save
res.putpalette(palette.ravel().tolist())
res.save('result.png')


Instruction: Pytorch .to('cuda') or .cuda() doesn't work and just get stuck

Input:
I am trying to do pytorch tutorial. When I try to set their device as cuda, it does not work and my code running get stuck.

For specific information, I am using conda environment of
python 3.7.3
pytorch 1.3.0
cuda 10.2 (NVIDIA RTX2080TI)

>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.device_count()
1
>>> torch.cuda.current_device()
0
>>> device = torch.device('cuda:0')
>>> device
device(type='cuda', index=0)
>>> aa = torch.randn(5)
>>> aa = tensor([-2.2084, -0.2700, 0.0921, -1.7678, 0.7642])
>>> aa.to(device)

nothing happens...

Can anybody please help me how to overcome this problem?

Output:
This has happened with the Pytorch 1.3.0 release (the release was this week). I too face this bug. Basically, when I call .to(device), it just hangs and does nothing.

If you would like to fix this temporarily, you can downgrade to PyTorch 1.2.0. To do this, I ran:

conda install pytorch=1.2.0 torchvision cudatoolkit=10.2 -c pytorch

I would have just commented but I do not have enough reputation to do that.


Instruction: Why is stft(istft(x)) ≠ x?

Input:
Why is stft(istft(x)) ≠ x?

Using PyTorch, I have computed the short-time Fourier transform of the inverse short-time Fourier transform of a tensor.
I have done so as shown below given a tensor x.
For x, the real and imaginary part are equal, or the imaginary part is set to zero -- both produces the same problem.

torch.stft(torchaudio.functional.istft(x, n_fft), n_fft)

As shown in the image, only a single one of the stripes in the tensor remains after applying stft(istft(x)) -- all other stripes vanish.
If stft(istft(x)) (bottom) was equal to x (top), both images would look similar.
Why are they so different?
It seems like stft(istft(x)) can pick up only certain frequencies of x.

x (top) and stft of istft of x (bottom)

I have also tried the same with scipy.signal.istft and scipy.signal.stft which causes the same issue.
Moreover, I have tried it with a wide range of tensors x, e.g., different randomized distributions, images, and other stripes.
Also, I have tried a variety of hyper-parameters for stft/istft.
Only for x generated by a short-time Fourier transform from a sound wave, it works.

Output:
A short-time Fourier transform produces a more data than there is in the original signal.  Where a signal has N real samples, then STFT might have 4N complex samples -- 8 times more data.

It follows that the ISTFT operation must discard 7/8 of the data you provide it.

Most of the data in a STFT is redundant, and if you just make up values for all of the data, it is unlikely to correspond to a real signal.

In that case, an implementation of ISTFT will probably use a least-squares fit or other method of producing a signal with an STFT that matches your data as closely as possible, but it won't always be close.


Instruction: How to read pictures from a big folder and split it into train, validation and test sets?

Input:
I am working on sign language gesture classifier using pytorch, I have pictures resembling each letter residing in a folder titled with that specific letter. E.g. folder "A" has "1_A_1.jpg", "1_A_2.jpg", "21_A_3.jpg".. etc.

I am trying to build a function that:

Iterates through the different folders
Splits the data into training, validation, and test sets
Labels those pictures with their respective folder name (i.e. letter label)
Returns 3 created folders that are train, test and validation

All the online code shows examples of splitting data coming from torchvision data sets (built in data sets), nothing from scratch.

I found the following on stackoverflow:

import os
import numpy as np
import argparse

def get_files_from_folder(path):

    files = os.listdir(path)
    return np.asarray(files)

def main(path_to_data, path_to_test_data, train_ratio):
    # get dirs
    _, dirs, _ = next(os.walk(path_to_data))

    # calculates how many train data per class
    data_counter_per_class = np.zeros((len(dirs)))

    for i in range(len(dirs)):
        path = os.path.join(path_to_data, dirs[i])
        files = get_files_from_folder(path)
        data_counter_per_class[i] = len(files)
    test_counter = np.round(data_counter_per_class * (1 - train_ratio))

    # transfers files
    for i in range(len(dirs)):
        path_to_original = os.path.join(path_to_data, dirs[i])
        path_to_save = os.path.join(path_to_test_data, dirs[i])

        #creates dir
        if not os.path.exists(path_to_save):
            os.makedirs(path_to_save)
        files = get_files_from_folder(path_to_original)
        # moves data
        for j in range(int(test_counter[i])):
            dst = os.path.join(path_to_save, files[j])
            src = os.path.join(path_to_original, files[j])
            shutil.move(src, dst)

and when I tried doing the following:

path_to_data= r'path\A'
path_to_test_data=r"path\test"
train_ratio=0.8

main(path_to_data,path_to_test_data,train_ratio)

Nothing really happened..

If I can get this working for train and test, I can easily extend it for validation.

Output:
Give this a go:

from pathlib import Path

def main(data_path, out_path, train_ratio):
    #1
    dir_paths = [child for child in Path(data_path).iterdir() if child.is_dir()]

    for i, dir_path in enumerate(dir_paths):
        #2
        files = list(dir_path.iterdir())
        train_len = int(len(files) * (1 - train_ratio))

        #3
        out_dir = Path(out_path).joinpath(dir_path.name)
        if not out_dir.exists():
            out_dir.mkdir(parents=True)

        #4
        for file_ in files[:train_len]:
            file_.replace(out_dir.joinpath(file_.name))

if __name__ == '__main__':
    main('data', 'test', 0.8)


Instruction: Deploy PyTorch model to GCP resulting in memory limit exceeded

Input:
I have trained a language model using transformer-lm which uses PyTorch. I would like to deploy the resulting model to the Google Cloud Platform as a Cloud Function. Cloud Functions are limited to 2 GB of memory.

The problem is that loading the model leads to an error as too much memory is used (memory limit exceeded). The model.pt file is 1.32 GB, and I use

torch.load(model_path / 'model.pt', map_location='cpu')

to load the model. Is there a way to i) compress the model? ii) not load the full model at once? or any other possibility to make it run on GCP?

Output:
Cloud Functions run their own server instances when triggered. If a called function is not used after some time, that instance is terminated. However, if you call it again while the instance is still running, the same instance along with the elements in your execution environment will be used again. This can cause your function crash.

In order to avoid that, you might want to implement a Garbage Collector. In Python, you can use the gc module for that. In particular, you can try the function gc.collect() to clear the memory.


Instruction: Convert pytorch tensor to numpy, and reshape

Input: I have a pytorch tensor [100, 1, 32, 32] corresponding to batch size of 100 images, 1 channel, height 32 and width 32. I want to reshape this tensor to have dimension [32*10, 32*10], such that the images are represented as a 10x10 grid, with the first 10 images on row 1, and so on. How to achieve this?

Output:
Update

More efficient and shorter version. To avoid using for-loop, we can permute a first.

import torch
a = torch.arange(9*2*2).view(9,1,2,2)
b = a.permute([0,1,3,2])
torch.cat(torch.split(b, 3),-1).view(6,6).t()
# tensor([[ 0,  1,  4,  5,  8,  9],
#         [ 2,  3,  6,  7, 10, 11],
#         [12, 13, 16, 17, 20, 21],
#         [14, 15, 18, 19, 22, 23],
#         [24, 25, 28, 29, 32, 33],
#         [26, 27, 30, 31, 34, 35]])

Original Answer

You can use torch.split and torch.cat to implement it.

import torch
a = torch.arange(9*2*2).view(9,1,2,2)

Assuming we have a tensor, which is a mini version of your original tensor. And it looks like,

tensor([[[[ 0,  1],
          [ 2,  3]]],
        [[[ 4,  5],
          [ 6,  7]]],
        [[[ 8,  9],
          [10, 11]]],
        [[[12, 13],
          [14, 15]]],
        [[[16, 17],
          [18, 19]]],
        [[[20, 21],
          [22, 23]]],
        [[[24, 25],
          [26, 27]]],
        [[[28, 29],
          [30, 31]]],
        [[[32, 33],
          [34, 35]]]])

Each 2x2 sub-matrix can be seen as one image. What you want to do is stacking the first three images to one row, next three images to the second row, and last three images to the third row. The "row" has actually two dim due to the 2x2 sub-matrix.

three_parts = torch.split(a,3)

torch.cat(torch.split(three_parts[0],1), dim=-1)

#tensor([[[[ 0,  1,  4,  5,  8,  9],
#          [ 2,  3,  6,  7, 10, 11]]]])

Here we only take the first part.

torch.cat([torch.cat(torch.split(three_parts[i],1),-1) for i in range(3)],0).view(6,6)
# tensor([[ 0,  1,  4,  5,  8,  9],
#         [ 2,  3,  6,  7, 10, 11],
#         [12, 13, 16, 17, 20, 21],
#         [14, 15, 18, 19, 22, 23],
#         [24, 25, 28, 29, 32, 33],
#         [26, 27, 30, 31, 34, 35]])


Instruction: How do I merge 2D Convolutions in PyTorch?

Input:
From linear algebra we know that linear operators are associative.
In the deep learning world, this concept is used to justify the introduction of non-linearities between NN layers, to prevent a phenomenon colloquially known as linear lasagna, (reference).
In signal processing this also leads to a well known trick to optimize memory and/or runtime requirements (reference).
So merging convolutions is a very useful tool from different perspectives. How to implement it with PyTorch?

Output:
If we have y = x * a * b (where * means convolution and a, b are your kernels), we can define c = a * b such that y = x * c = x * a * b as follows:

import torch

def merge_conv_kernels(k1, k2):
    """
    :input k1: A tensor of shape ``(out1, in1, s1, s1)``
    :input k1: A tensor of shape ``(out2, in2, s2, s2)``
    :returns: A tensor of shape ``(out2, in1, s1+s2-1, s1+s2-1)``
      so that convolving with it equals convolving with k1 and
      then with k2.
    """
    padding = k2.shape[-1] - 1
    # Flip because this is actually correlation, and permute to adapt to BHCW
    k3 = torch.conv2d(k1.permute(1, 0, 2, 3), k2.flip(-1, -2),
                      padding=padding).permute(1, 0, 2, 3)
    return k3

To illustrate the equivalence, this example combines two kernels with 900 and 5000 parameters respectively into an equivalent kernel of 28 parameters:

# Create 2 conv. kernels
out1, in1, s1 = (100, 1, 3)
out2, in2, s2 = (2, 100, 5)
kernel1 = torch.rand(out1, in1, s1, s1, dtype=torch.float64)
kernel2 = torch.rand(out2, in2, s2, s2, dtype=torch.float64)

# propagate a random tensor through them. Note that padding
# corresponds to the "full" mathematical operation (s-1)
b, c, h, w = 1, 1, 6, 6
x = torch.rand(b, c, h, w, dtype=torch.float64) * 10
c1 = torch.conv2d(x, kernel1, padding=s1 - 1)
c2 = torch.conv2d(c1, kernel2, padding=s2 - 1)

# check that the collapsed conv2d is same as c2:
kernel3 = merge_conv_kernels(kernel1, kernel2)
c3 = torch.conv2d(x, kernel3, padding=kernel3.shape[-1] - 1)
print(kernel3.shape)
print((c2 - c3).abs().sum() < 1e-5)

Note: The equivalence is assuming that we have unlimited numerical resolution. I think there was research on stacking many low-resolution-float linear operations and showing that the networks profited from numerical error, but I am unable to find it. Any reference is appreciated!


Instruction: How to write custom CrossEntropyLoss

Input:
I am learning Logistic Regression within Pytorch and to better understand I am defining a custom CrossEntropyLoss as below:

def softmax(x):
    exp_x = torch.exp(x)
    sum_x = torch.sum(exp_x, dim=1, keepdim=True)

    return exp_x/sum_x

def log_softmax(x):
    return torch.exp(x) - torch.sum(torch.exp(x), dim=1, keepdim=True)

def CrossEntropyLoss(outputs, targets):
    num_examples = targets.shape[0]
    batch_size = outputs.shape[0]
    outputs = log_softmax(outputs)
    outputs = outputs[range(batch_size), targets]

    return - torch.sum(outputs)/num_examples

I also make my own logistic regression (to predict FashionMNIST) as below:

input_dim = 784 # 28x28 FashionMNIST data
output_dim = 10

w_init = np.random.normal(scale=0.05, size=(input_dim,output_dim))
w_init = torch.tensor(w_init, requires_grad=True).float()
b = torch.zeros(output_dim)

def my_model(x):
    bs = x.shape[0]
    return x.reshape(bs, input_dim) @ w_init + b

To validate my custom crossentropyloss, I compared it with nn.CrossEntropyLoss from Pytorch by applying it on FashionMNIST data as below:

criterion = nn.CrossEntropyLoss()

for X, y in trn_fashion_dl:
    outputs = my_model(X)
    my_outputs = softmax(outputs)

    my_ce = CrossEntropyLoss(my_outputs, y)
    pytorch_ce = criterion(outputs, y)

    print (f'my custom cross entropy: {my_ce.item()}\npytorch cross entroopy: {pytorch_ce.item()}')
    break

My question is toward the results my_ce (my cross entropy) vs pytorch_ce (pytorch cross entropy) where they are different:

my custom cross entropy: 9.956839561462402
pytorch cross entroopy: 2.378990888595581

I appreciate your help in advance!

Output:
There are two bugs in your code.

The log_softmax(x) should be like,

def log_softmax(x):
    return torch.log(softmax(x))

When you calculate your own CE loss, you should input outputs instead of my_outputs. Because you will calculate softmax inside your own CE loss function. It should be like,

outputs = my_model(X)
my_ce = CrossEntropyLoss(outputs, y)
pytorch_ce = criterion(outputs, y)

Then you will have identical results.

my custom cross entropy: 3.584486961364746
pytorch cross entroopy: 3.584486961364746


Instruction: Resnet-18 as backbone in Faster R-CNN

Input:
I code with pytorch and I want to use resnet-18 as backbone of Faster R-RCNN. When I print structure of resnet18, this is the output:

>>import torch
>>import torchvision
>>import numpy as np
>>import torchvision.models as models

>>resnet18 = models.resnet18(pretrained=False)
>>print(resnet18)

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=512, out_features=1000, bias=True)
)

My question is, until which layer it is feature extractor?  is AdaptiveAvgPool2d should be part of backbone of Faster R-CNN?

In this toturial, it is shown how to train a Mask R-CNN with an arbitrary backbone, I want to do the same thing with Faster R-CNN and train a Faster R-CNN with resnet-18 but until which layer should be part of feature extractor is confusing for me.

I know how to use resnet+Feature Pyramid Network as backbone, My question is about resent.

Output:
If we want to use output of Adaptive Average Pooling we use this code for different Resnets:

# backbone
        if backbone_name == 'resnet_18':
            resnet_net = torchvision.models.resnet18(pretrained=True)
            modules = list(resnet_net.children())[:-1]
            backbone = nn.Sequential(*modules)
            backbone.out_channels = 512
        elif backbone_name == 'resnet_34':
            resnet_net = torchvision.models.resnet34(pretrained=True)
            modules = list(resnet_net.children())[:-1]
            backbone = nn.Sequential(*modules)
            backbone.out_channels = 512
        elif backbone_name == 'resnet_50':
            resnet_net = torchvision.models.resnet50(pretrained=True)
            modules = list(resnet_net.children())[:-1]
            backbone = nn.Sequential(*modules)
            backbone.out_channels = 2048
        elif backbone_name == 'resnet_101':
            resnet_net = torchvision.models.resnet101(pretrained=True)
            modules = list(resnet_net.children())[:-1]
            backbone = nn.Sequential(*modules)
            backbone.out_channels = 2048
        elif backbone_name == 'resnet_152':
            resnet_net = torchvision.models.resnet152(pretrained=True)
            modules = list(resnet_net.children())[:-1]
            backbone = nn.Sequential(*modules)
            backbone.out_channels = 2048
        elif backbone_name == 'resnet_50_modified_stride_1':
            resnet_net = resnet50(pretrained=True)
            modules = list(resnet_net.children())[:-1]
            backbone = nn.Sequential(*modules)
            backbone.out_channels = 2048
        elif backbone_name == 'resnext101_32x8d':
            resnet_net = torchvision.models.resnext101_32x8d(pretrained=True)
            modules = list(resnet_net.children())[:-1]
            backbone = nn.Sequential(*modules)
            backbone.out_channels = 2048

If we want to use convolution feature map we use this code:

 # backbone
        if backbone_name == 'resnet_18':
            resnet_net = torchvision.models.resnet18(pretrained=True)
            modules = list(resnet_net.children())[:-2]
            backbone = nn.Sequential(*modules)

        elif backbone_name == 'resnet_34':
            resnet_net = torchvision.models.resnet34(pretrained=True)
            modules = list(resnet_net.children())[:-2]
            backbone = nn.Sequential(*modules)

        elif backbone_name == 'resnet_50':
            resnet_net = torchvision.models.resnet50(pretrained=True)
            modules = list(resnet_net.children())[:-2]
            backbone = nn.Sequential(*modules)

        elif backbone_name == 'resnet_101':
            resnet_net = torchvision.models.resnet101(pretrained=True)
            modules = list(resnet_net.children())[:-2]
            backbone = nn.Sequential(*modules)

        elif backbone_name == 'resnet_152':
            resnet_net = torchvision.models.resnet152(pretrained=True)
            modules = list(resnet_net.children())[:-2]
            backbone = nn.Sequential(*modules)

        elif backbone_name == 'resnet_50_modified_stride_1':
            resnet_net = resnet50(pretrained=True)
            modules = list(resnet_net.children())[:-2]
            backbone = nn.Sequential(*modules)

        elif backbone_name == 'resnext101_32x8d':
            resnet_net = torchvision.models.resnext101_32x8d(pretrained=True)
            modules = list(resnet_net.children())[:-2]
            backbone = nn.Sequential(*modules)


Instruction: Does a `DataLoader` created from `ConcatDataset` create a batch from a different files, or a single file?

Input:
I am working with multiple files, and multiple training samples in each file. I will use ConcatDataset as described here:

https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/7

I need to have negative samples in addition to my true samples, and I need my negative samples to be randomly selected from all the training data files. So, I am wondering, would the returned batch samples just be a random consecutive chuck from a single file, or would be batch span across multiple random indexes across all the datafiles?

If there are more details needed about what I am trying to do exactly, it's because I am trying to train over a TPU with Pytorch XLA.

Normally for negative samples, I would just use a 2nd DataSet and DataLoader, however, I am trying to train over TPUs with Pytorch XLA (alpha was just released a few days ago https://github.com/pytorch/xla ), and to do that I need to send my DataLoader to a torch_xla.distributed.data_parallel.DataParallel object, like model_parallel(train_loop_fn, train_loader) which can be seen in these example notebooks

https://github.com/pytorch/xla/blob/master/contrib/colab/resnet18-training-xrt-1-15.ipynb

https://github.com/pytorch/xla/blob/master/contrib/colab/mnist-training-xrt-1-15.ipynb

So, I am now limited to a single DataLoader, which will need to handle both the true samples, and negative samples that need to be randomly selected from all my files.

Output:
ConcatDataset is a custom class that is subclassed from torch.utils.data.Dataset. Let's take a look at one example.

class ConcatDataset(torch.utils.data.Dataset):
    def __init__(self, *datasets):
        self.datasets = datasets

    def __getitem__(self, i):
        return tuple(d[i] for d in self.datasets)

    def __len__(self):
        return min(len(d) for d in self.datasets)

train_loader = torch.utils.data.DataLoader(
             ConcatDataset(
                 dataset1,
                 dataset2
             ),
             batch_size=args.batch_size,
             shuffle=True,
             num_workers=args.workers,
             pin_memory=True)

for i, (input, target) in enumerate(train_loader):
    ...

Here, two datasets namely dataset1 (a list of examples) and dataset2 are combined to form a single training dataset. The __getitem__ function returns one example from the dataset and will be used by the BatchSampler to form the training mini-batches.

  Would the returned batch samples just be a random consecutive chuck from a single file, or would be batch span across multiple random indexes across all the datafiles?

Since you have combined all your data files to form one dataset, now it depends on what BatchSampler do you use to sample mini-batches. There are several samplers implemented in PyTorch, for example, RandomSampler, SequentialSampler, SubsetRandomSampler, WeightedRandomSampler. See their usage in the documentation.

You can have your custom BatchSampler too as follows.

class MyBatchSampler(Sampler):
    def __init__(self, *params):
        # write your code here

    def __iter__(self):
        # write your code here
        # return an iterable

    def __len__(self):
        # return the size of the dataset

The __iter__ function should return an iterable of mini-batches. You can implement your logic of forming mini-batches in this function.

To randomly sample negative examples for training, one alternative could be to pick negative examples for each positive example in the __init__ function of the ConcatDataset class.


Instruction: RuntimeError: size mismatch, m1: [32 x 1], m2: [32 x 9]

Input:
I'm building a CNN and training it on hand sign gesture classification for letters A through I (9 classes), each image is RGB with 224x224 size.

Not sure which matrix I need to transpose and how. I have managed to match the inputs and outputs of layers, but that matrix multiplication thing, not really sure how to fix it.

class LargeNet(nn.Module):
    def __init__(self):
        super(LargeNet, self).__init__()
        self.name = "large"
        self.conv1 = nn.Conv2d(3, 5, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(5, 10, 5)
        self.fc1 = nn.Linear(10 * 53 * 53, 32)
        self.fc2 = nn.Linear(32, 9)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        print('x1')
        x = self.pool(F.relu(self.conv2(x)))
        print('x2')
        x = x.view(-1, 10*53*53)
        print('x3')
        x = F.relu(self.fc1(x))
        print('x4')
        x = x.view(-1, 1)
        x = self.fc2(x)
        print('x5')
        x = x.squeeze(1) # Flatten to [batch_size]
        return x

and training code

#Loss and optimizer
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.SGD(model2.parameters(), lr=learning_rate, momentum=0.9)

# Train the model
total_step = len(train_loader)
loss_list = []
acc_list = []
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        print(i,images.size(),labels.size())
        # Run the forward pass
        outputs = model2(images)
        labels=labels.unsqueeze(1)
        labels=labels.float()
        loss = criterion(outputs, labels)

The code prints up to x4 and then I get this error RuntimeError: size mismatch, m1: [32 x 1], m2: [32 x 9] at C:\w\1\s\tmp_conda_3.7_055457\conda\conda-bld\pytorch_1565416617654\work\aten\src\TH/generic/THTensorMath.cpp:752

Complete traceback error: https://ibb.co/ykqy5wM

Output:
You don't need x=x.view(-1,1) and x = x.squeeze(1) in your forward function. Remove these two lines. Your output shape would be (batch_size, 9).

Also, you need to convert labels to one-hot encoding, which is in shape of (batch_size, 9).

class LargeNet(nn.Module):
    def __init__(self):
        super(LargeNet, self).__init__()
        self.name = "large"
        self.conv1 = nn.Conv2d(3, 5, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(5, 10, 5)
        self.fc1 = nn.Linear(10 * 53 * 53, 32)
        self.fc2 = nn.Linear(32, 9)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 10*53*53)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

model2 = LargeNet()
#Loss and optimizer
criterion = nn.BCEWithLogitsLoss()
# nn.BCELoss()
optimizer = optim.SGD(model2.parameters(), lr=0.1, momentum=0.9)

images = torch.from_numpy(np.random.randn(2,3,224,224)).float() # fake images, batch_size is 2
labels = torch.tensor([1,2]).long() # fake labels

outputs = model2(images)
one_hot_labels = torch.eye(9)[labels]
loss = criterion(outputs, one_hot_labels)


Instruction: Listing first and second values in a tensor

Input:
I know this might be a simple problem for some of you. I am not able to get the output as expected.
The input tensor is given below.

X = tensor([[50.7500, 44.0000],
        [47.0000, 47.0000],
        [42.5000, 52.2500],
        [59.6163, 50.7097],
        [54.6682, 54.6682],
        [48.7304, 61.5956],
        [71.3156, 59.5631],
        [64.7864, 64.7864],
        [56.9515, 73.9272]])

I want two tensors A and B such that

A = tensor([[50.7500],
        [47.0000],
        [42.5000],
        [59.6163],
        [54.6682],
        [48.7304],
        [71.3156],
        [64.7864],
        [56.9515]]) and
B = tensor([[44.0000],
        [47.0000],
        [52.2500],
        [50.7097],
        [54.6682],
        [61.5956],
        [59.5631],
        [64.7864],
        [73.9272]])

This is what I tried till now.

idx = torch.LongTensor([0])
idx1 = torch.LongTensor([1])
A = X.index_select(1,idx)
B = X.index_select(1,idx1)

which is not giving me output as expected. Thanks in advance.

Output:
You want to pick elements from second index in the tensor, you can do this as follows:

A=X[:,0]
B=X[:,1]

The output of this will be :

print(B)
>>>tensor([44.0000, 47.0000, 52.2500, 50.7097, 54.6682, 61.5956, 59.5631,
64.7864, 73.9272])

print(A)
>>>tensor([50.7500, 47.0000, 42.5000, 59.6163, 54.6682, 48.7304,
71.3156, 64.7864,56.9515])

But since you want two dimensional tensors, you can then do

A.unsqueeze(dim=1)
B.unsqueeze(dim=1)

In above case you had to explicitly tell which dimension you have to add/unsqueeze but you can avoid that by using list of indexes as below:

A = X[:, 0:1]
B = x[:, 1:2]

Using indexes of type i:i+1 will preserve the number dimensions from old tensor.


Instruction: Correct way to use custom weight maps in unet architecture

Input:
There is a famous trick in u-net architecture to use custom weight maps to increase accuracy. Below are the details of it:

Now, by asking here and at multiple other place, I get to know about 2 approaches. I want to know which one is correct or is there any other right approach which is more correct?

First is to use torch.nn.Functional method in the training loop:
loss = torch.nn.functional.cross_entropy(output, target, w) where w will be the calculated custom weight.

Second is to use reduction='none' in the calling of loss function outside the training loop
criterion = torch.nn.CrossEntropy(reduction='none')
and then in the training loop multiplying with the custom weight:
gt # Ground truth, format torch.long
pd # Network output
W # per-element weighting based on the distance map from UNet
loss = criterion(pd, gt)
loss = W*loss # Ensure that weights are scaled appropriately
loss = torch.sum(loss.flatten(start_dim=1), axis=0) # Sums the loss per image
loss = torch.mean(loss) # Average across a batch

Now, I am kinda confused which one is right or is there any other way, or both are right?

Output:
The weighting portion looks like just simply weighted cross entropy which is performed like this for the number of classes (2 in the example below).

weights = torch.FloatTensor([.3, .7])
loss_func = nn.CrossEntropyLoss(weight=weights)

EDIT:

Have you seen this implementation from Patrick Black?

# Set properties
batch_size = 10
out_channels = 2
W = 10
H = 10

# Initialize logits etc. with random
logits = torch.FloatTensor(batch_size, out_channels, H, W).normal_()
target = torch.LongTensor(batch_size, H, W).random_(0, out_channels)
weights = torch.FloatTensor(batch_size, 1, H, W).random_(1, 3)

# Calculate log probabilities
logp = F.log_softmax(logits)

# Gather log probabilities with respect to target
logp = logp.gather(1, target.view(batch_size, 1, H, W))

# Multiply with weights
weighted_logp = (logp * weights).view(batch_size, -1)

# Rescale so that loss is in approx. same interval
weighted_loss = weighted_logp.sum(1) / weights.view(batch_size, -1).sum(1)

# Average over mini-batch
weighted_loss = -1. * weighted_loss.mean()


Instruction: Convert list of two dimensional DataFrame to Torch Tensor

Input:
Goal: I am working with RNNs in PyTorch, and my data is given by a list of DataFrames, where each DataFrame means one observation like:

import numpy as np
data = [pd.DataFrame(np.zeros((5,50))) for x in range(100)]

which means 100 observation, with 50 parameters and 5 timesteps each. For my Model i need a tensor of shape (100,5,50).

Issue: I tried a lot of things but nothing seems to work, does anyone know how this is done?
This approaches doesn't work:

import torch
torch.tensor(np.array(data))

I thing the problem is to convert the DataFrames into Arrays and the List into a Tensor at the same time.

Output:
I don't think you can convert the list of dataframes in a single command, but you can convert the list of dataframes into a list of tensors and then concatenate the list.

E.g.

import pandas as pd
import numpy as np
import torch

data = [pd.DataFrame(np.zeros((5,50))) for x in range(100)]

list_of_arrays = [np.array(df) for df in data]
torch.tensor(np.stack(list_of_arrays))

#or

list_of_tensors = [torch.tensor(np.array(df)) for df in data]
torch.stack(list_of_tensors)


Instruction: Use SHAP Values for PyTorch RNN / LSTM

Input:
Is there a way to do the above?
The SHAP Package is very helpful and works pretty well for PyTorch Neural Nets. For PyTorch RNNs i get the error message below (for LSTMs its the same):

Seems like it doesn't work but is there a workaround or something? Does anyone have experience with PyTorch and SHAP?

Output:
'RNNs aren't yet supported for the PyTorch DeepExplainer (A warning pops up to let you know which modules aren't supported yet: Warning: unrecognized nn.Module: RNN). In this case, the explainer assumes the module is linear, and makes no change to the gradient. Since RNNs contain nonlinearities, this is probably contributing to the problem.' That was an answer I found at Shap.

Try to check captum.ai that is built on PyTorch.


Instruction: How to vectorize custom algorithms in numpy or pytorch?

Input:
Suppose I have two matrices:

A: size k x m

B: size m x n

Using a custom operation, my output will be k x n.

This custom operation is not a dot product between the rows of A and columns of B. Suppose this custom operation is defined as:

For the Ith row of A and Jth column of B, the i,j element of the output is:

sum( (a[i] + b[j]) ^20 ), i loop over I, j loops over J

The only way I can see to implement this is to expand this equation, calculate each term, them sum them.

Is there a way in numpy or pytorch to do this without expanding the equation?

Output:
Apart from the method @hpaulj outlines in the comments, you can also use the fact that what you are calculating is essentially a pair-wise Minkowski distance:

import numpy as np
from scipy.spatial.distance import cdist

k,m,n = 10,20,30
A = np.random.random((k,m))
B = np.random.random((m,n))

method1 = ((A[...,None]+B)**20).sum(axis=1)
method2 = cdist(A,-B.T,'m',p=20)**20

np.allclose(method1,method2)
# True


Instruction: Indices out of range for MaxUnpool2d

Input:
I am trying to understand unpooling in Pytorch because I want to build a convolutional auto-encoder.

I have the following code

from torch.autograd import Variable

data = Variable(torch.rand(1, 73, 480))
pool_t = nn.MaxPool2d(2, 2, return_indices=True)
unpool_t = nn.MaxUnpool2d(2, 2)
out, indices1 = pool_t(data)
out = unpool_t(out, indices1)

But I am constantly getting this error on the last line (unpooling).

IndexError: tuple index out of range

Although the data is simulated in this example, the input has to be of that shape because of the preprocessing that has to be done.

I am fairly new to convolutional networks, but I have even tried using a ReLU and convolutional 2D layer before the pooling however, the indices always seem to be incorrect when unpooling for this shape.

Output:
Your data is 1D and you are using 2D pooling and unpooling operations.

PyTorch interpret the first two dimensions of tensors as "batch dimension" and "channel"/"feature space" dimension. The rest of the dimensions are treated as spatial dimensions.
So, in your example, data is 3D tensor of size (1, 73, 480) and is interpret by pytorch as a single batch ("batch dimension" = 1) with 73 channels per sample and 480 samples.
For some reason MaxPool2d works for you and treats the channel dimension as a spatial dimension and sample this as well - I'm not sure this is a bug or a feature.

If you do want to sample along the second dimension you can add an additional dimension, making data a 4D tensor:

out, indices1 = pool_t(data[None,...])
In [11]: out = unpool_t(out, indices1, data[None,...].size())


Instruction: How to load the saved tokenizer from pretrained model

Input:
I fine-tuned a pretrained BERT model in Pytorch using huggingface transformer. All the training/validation is done on a GPU in cloud.
At the end of the training, I save the model and tokenizer like below:
best_model.save_pretrained('./saved_model/')
tokenizer.save_pretrained('./saved_model/')

This creates below files in the saved_model directory:
config.json
added_token.json
special_tokens_map.json
tokenizer_config.json
vocab.txt
pytorch_model.bin

Now, I download the saved_model directory in my computer and want to load the model and tokenizer. I can load the model like below
model = torch.load('./saved_model/pytorch_model.bin',map_location=torch.device('cpu'))
But how do I load the tokenizer? I am new to pytorch and not sure because there are multiple files. Probably I am not saving the model in the right way?

Output:
If you look at the syntax, it is the directory of the pre-trained model that you are supposed to pass. Hence, the correct way to load tokenizer must be:
tokenizer = BertTokenizer.from_pretrained(<Path to the directory containing pretrained model/tokenizer>)

In your case:
tokenizer = BertTokenizer.from_pretrained('./saved_model/')

./saved_model here is the directory where you'll be saving your pretrained model and tokenizer.


Instruction: pytorch - gradients not calculated for parameters

Input:
a = torch.nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float, device=device))
b = torch.nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float, device=device))
c = a + 1
d = torch.nn.Parameter(c, requires_grad=True,)
for epoch in range(n_epochs):
    yhat = d + b * x_train_tensor
    error = y_train_tensor - yhat
    loss = (error ** 2).mean()
    loss.backward()
    print(a.grad)
    print(b.grad)
    print(c.grad)
    print(d.grad)

Prints out

None
tensor([-0.8707])
None
tensor([-1.1125])

How do I learn the gradient for a and c? variable d needs to stay a parameter

Output:
Basically, when you create a new tensor, like torch.nn.Parameter() or torch.tensor(), you are creating a leaf node tensor.

And when you do something like c=a+1, c will be intermediate node. You can print(c.is_leaf) to check whether the tensor is leaf node or not. Pytorch will not calculate the gradient of intermediate node in default.

In your code snippet, a, b, d are all leaf node tensor, and c is intermediate node. c.grad will None as pytorch doesn't calculate the gradient for intermediate node. a is isolated from the graph when you call loss.backword(). That's why a.grad is also None.

If you change the code to this

a = torch.nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float, device=device))
b = torch.nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float, device=device))
c = a + 1
d = c
for epoch in range(n_epochs):
    yhat = d + b * x_train_tensor
    error = y_train_tensor - yhat
    loss = (error ** 2).mean()
    loss.backward()
    print(a.grad) # Not None
    print(b.grad) # Not None
    print(c.grad) # None
    print(d.grad) # None

You will find a and b have gradients, but c.grad and d.grad are None, because they're intermediate node.


Instruction: How to fetch image dataset from Google Drive to Colab?

Input:
I have this very weird problem. I have searched across internet, read documentation but am not able to figure out how to do it. So what I want to do is train a classifier using Colab. And for that I have a image dataset of dogs on my local machine.

So what I did was I packed that dataset folder of images into a zip file and uploaded it onto Drive. Then from Colab I mounted the drive and from there I tried to unzip the files. Everything good. But I've realised that after sometime some of the extracted files get deleted. And thing is that those files aren't on Colab storage, but instead on Drive and I dunno why they are getting deleted after sometime. Like about an hour.

So far I've used the following commands to do the extraction -

from google.colab import drive
drive.mount('/content/drive')

from zipfile import ZipFile
filename = 'Stanford Dogs Dataset.zip'

with ZipFile(filename, 'r') as zip:
  zip.extractall()
  print('Done')

and also tried this -

!unzip filename -d destination

Not sure where I am going wrong. And also, dunno why the extracted files though being extracted to a subfolder within drive, also starts showing up on the main root directory. And no I am not talking about the recent section, because when I want to check their location then they points to the root of the drive. It's all so confusing.

Output:
First you mount google drive

from google.colab import drive
drive.mount('/gdrive')

Then you can copy from your drive using !cp

!cp '/gdrive/My Drive/my_file' 'my_file'

then you can work as in your pc, unzip and ...


Instruction: Pytorch - going back and forth between eval() and train() modes

Input:
I'm studying "Deep Reinforcement Learning" and build my own example after pytorch's REINFORCEMENT LEARNING (DQN) TUTORIAL.

I'm implement actor's strategy as follows:
1. model.eval()
2. get best action from a model
3. self.net.train()

The question is:
Does going back and forth between eval() and train() modes cause any damage to optimization process?

The model includes only Linear and BatchNorm1d layers.
As far as I know when using BatchNorm1d one must perform model.eval() to use a model, because there is different results in eval() and train() modes.

When training Classification Neural Network the model.eval() performed only after training is finished, but in case of "Deep Reinforcement Learning" it is usual to use strategy and then continue the optimization process.

I'm wondering if going back and forth between modes is "harmless" to optimization process?

def strategy(self, state):
    # Explore or Exploit
    if self.epsilon > random():
        action = choice(self.actions)
    else:
        self.net.eval()
        action = self.net(state.unsqueeze(0)).max(1)[1].detach()
        self.net.train()

Output:
eval() puts the model in the evaluation mode.

In the evaluation mode, the Dropout layer just acts as a "passthrough" layer.
During training, a BatchNorm layer keeps a running estimate of its computed mean and variance. The running sum is kept with a default momentum of 0.1. During the evaluation, this running mean/variance is used for normalization.

So, going back and forth between eval() and train() modes do not cause any damage to the optimization process.


Instruction: Getting Torch to recognize GPU

Input:
How do you get Torch to recognize CUDA on your video card?

I have a Nvidia GeForce GT 1030 running under Ubuntu 18.04, and it claims to support CUDA, yet when I first tested Torch with it by running:

virtualenv -p python3.7 .env
. .env/bin/activate
pip install torch
python -c "import torch; print(torch.cuda.is_available())"

it returned False, along with the warning:

    The NVIDIA driver on your system is too old (found version 9010).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.

So I ran all system updates and used Ubuntu's proprietary driver installer to install the most recent Nvidia-435 driver for my card.

However, torch.cuda.is_available() still returns false, but now it doesn't give me any warning.

Have I mis-configured Torch or does my GPU just not support CUDA?

Output: Nevermind. I spoke too soon. I didn't reboot after switching over the driver, and apparently that broke nvidia-smi and some other things that loaded the CUDA driver. After the reboot, Torch now recognizes CUDA 10.1 support.


Instruction: PyTorch BERT TypeError: forward() got an unexpected keyword argument 'labels'

Input:
Training a BERT model using PyTorch transformers (following the tutorial here).

Following statement in the tutorial

loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)

leads to

TypeError: forward() got an unexpected keyword argument 'labels'

Here is the full error,

TypeError                                 Traceback (most recent call last)
<ipython-input-53-56aa2f57dcaf> in <module>
     26         optimizer.zero_grad()
     27         # Forward pass
---> 28         loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
     29         train_loss_set.append(loss.item())
     30         # Backward pass

~/anaconda3/envs/systreviewclassifi/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

TypeError: forward() got an unexpected keyword argument 'labels'

I cant seem to figure out what kind of argument the forward() function expects.

There is a similar problem here, but I still do not get what the solution is.

System information:

OS: Ubuntu 16.04 LTS
Python version: 3.6.x
Torch version: 1.3.0
Torch Vision version: 0.4.1
PyTorch transformers version: 1.2.0

Output:
As far as I know, the BertModel does not take labels in the forward() function. Check out the forward function parameters.
I suspect you are trying to fine-tune the BertModel for sequence classification task and the API provides a class for that which is BertForSequenceClassification. As you can see its forward() function definition:
def forward(self, input_ids, attention_mask=None, token_type_ids=None,
            position_ids=None, head_mask=None, labels=None):

Please note, the forward() method returns the followings.
Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs:
        **loss**: (`optional`, returned when ``labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``:
            Classification (or regression if config.num_labels==1) loss.
        **logits**: ``torch.FloatTensor`` of shape ``(batch_size, config.num_labels)``
            Classification (or regression if config.num_labels==1) scores (before SoftMax).
        **hidden_states**: (`optional`, returned when ``config.output_hidden_states=True``)
            list of ``torch.FloatTensor`` (one for the output of each layer + the output of the embeddings)
            of shape ``(batch_size, sequence_length, hidden_size)``:
            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
        **attentions**: (`optional`, returned when ``config.output_attentions=True``)
            list of ``torch.FloatTensor`` (one for each layer) of shape ``(batch_size, num_heads, sequence_length, sequence_length)``:
            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Hope this helps!


Instruction: why the `test process` is inside the epoch loop in deep learning?

Input:
I am new to deep learning, here is the code I saw

The code is fine, but I don't understand the follow:

for epoch in range(1, args.epochs + 1):
    train(epoch)
    test(epoch)
    with torch.no_grad():
        sample = torch.randn(64, 20).to(device)
        sample = model.decode(sample).cpu()

In machine learning, when we finish training, we fix the model parameters for the test dataset. here are my two questions:

(1) In deep learning, we have the training, validation, test dataset. Is the code test(epoch) actually for validation set?  There we fix model parameters and predict for test dataset outside the epoch loop?

(2) what is with torch.no_grad(): mean? and why it also inside the epoch loop?

Thanks a lot.

Output:
Yes, the test(epoch) is actually for validation here (Update: not exactly validation, check below answer).
with torch.no_grad() means that you're switching off the gradients (required for backpropagation during training). In validation/testing you don't need them, and it'll save memory and computations. Read more here.

Also, check the tutorial here.


Instruction: Is there an equivalent function of pytorch named "index_select" in tensorflow

Input: I tried to translate pytorch code to tensorflow. So I wanna know is there an equivalent function of pytorch named index_select in tensorflow

Output:
I haven't found a similar api can directly achieve it, but we can use tf.slice to implement it.
def tf_index_select(input_, dim, indices):
    """
    input_(tensor): input tensor
    dim(int): dimension
    indices(list): selected indices list
    """
    shape = input_.get_shape().as_list()
    if dim == -1:
        dim = len(shape)-1
    shape[dim] = 1

    tmp = []
    for idx in indices:
        begin = [0]*len(shape)
        begin[dim] = idx
        tmp.append(tf.slice(input_, begin, shape))
    res = tf.concat(tmp, axis=dim)

    return res

Here is an example to show the equivalence.
import tensorflow as tf
import torch
import numpy as np

a = np.arange(2*3*4).reshape(2,3,4)
dim = 1
indices = [0,2]
# array([[[ 0,  1,  2,  3],
#         [ 4,  5,  6,  7],
#         [ 8,  9, 10, 11]],

#        [[12, 13, 14, 15],
#         [16, 17, 18, 19],
#         [20, 21, 22, 23]]])

# pytorch
res = torch.tensor(a).index_select(dim, torch.tensor(indices))
# tensor([[[ 0,  1,  2,  3],
#          [ 8,  9, 10, 11]],

#         [[12, 13, 14, 15],
#          [20, 21, 22, 23]]])

# tensorflow
res = tf_index_select(tf.constant(a), dim, indices)
# tensor([[[ 0,  1,  2,  3],
#          [ 8,  9, 10, 11]],

#         [[12, 13, 14, 15],
#          [20, 21, 22, 23]]])


Instruction: bool value of Tensor with more than one value is ambiguous Pytorch error

Input:
I have a Neural network that i want to train but i keep getting an "bool value of Tensor with more than one value is ambiguous" Error.

Here is my network:

from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader
import torch.nn as nn
from torch.nn import functional as F
from DataSetLoader import ReplaysDataSet

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.l1 = torch.nn.Linear(12,24)
        self.l2 = torch.nn.Linear(24, 20)
        self.l3 = torch.nn.Linear(20, 16)
        self.l4 = torch.nn.Linear(16, 10)
        self.l5 = torch.nn.Linear(10, 6)
        self.l6 = torch.nn.Linear(6, 1)

    def forward(self, t):
        t = F.relu(self.l1(t))
        t = F.relu(self.l2(t))
        t = F.relu(self.l3(t))
        t = F.relu(self.l4(t))
        t = F.relu(self.l5(t))
        y_pred = F.relu(self.l6(t))

        return y_pred
    def get_num_correct(self,preds,labels):
        return preds.argmax(dim=1).eq(labels).sum()

class Test():
    model = Model()
    optimiser = torch.optim.SGD(model.parameters(), lr=0.01)

    dataset = ReplaysDataSet()
    trainLoader = DataLoader(dataset=dataset, batch_size=250, shuffle=True)
    batch = next(iter(trainLoader))
    criterion = nn.MSELoss
    for epoch in range(10):
        totalLoss = 0
        totalCorrect = 0
        for batch in trainLoader:

            data, label = batch
            prediction = model(data)
            prediction = prediction.reshape(250)
            print(prediction)
            print(label)
            loss = criterion(prediction, label)
            optimiser.zero_grad()

            loss.backward()
            optimiser.step()

            totalLoss += loss.item()
            print(totalLoss)
            totalCorrect += model.get_num_correct(prediction, label)
            print(totalCorrect)

Here is my Data loader

import torch
from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader
from torch.nn import functional as F

class ReplaysDataSet(Dataset):
    def __init__(self):
        self.xy = np.genfromtxt("dataset.csv", delimiter=',', dtype=np.float32)
        self.length = len(self.xy)
        self.x_data = torch.from_numpy(self.xy[0:,1:])
        self.y_data = torch.from_numpy(self.xy[0:,0])
        self.length = len(self.xy)
    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]
    def __len__(self):
        return self.length

And here is some data from the csv im training on

1,303,497,784,748,743,169,462,479,785,310,26,701
1,658,598,645,786,248,381,80,428,248,530,591,145
0,38,490,796,637,130,380,226,359,720,392,464,497
0,94,752,645,801,381,479,475,381,227,645,445,248
0,59,806,254,521,91,538,212,645,609,227,545,531
1,65,254,685,565,91,796,445,658,465,485,472,184
1,385,248,211,612,82,38,485,652,212,373,563,26
1,796,596,785,310,145,479,142,685,748,635,798,474
1,380,658,485,645,36,598,806,428,786,798,645,113
0,743,214,625,642,530,784,645,641,65,598,786,637

The error i get is

Traceback (most recent call last):
  File "C:/Users/tayya/PycharmProjects/untitled/NetworkFile.py", line 32, in <module>
    class Test():
  File "C:/Users/tayya/PycharmProjects/untitled/NetworkFile.py", line 50, in Test
    loss = criterion(prediction, label)
File "C:\Users\tayya\PycharmProjects\untitled\venv\lib\site-packages\torch\nn\modules\loss.py", line 428, in __init__
super(MSELoss, self).__init__(size_average, reduce, reduction)
  File "C:\Users\tayya\PycharmProjects\untitled\venv\lib\site-packages\torch\nn\modules\loss.py", line 12, in __init__
    self.reduction = _Reduction.legacy_get_string(size_average, reduce)
  File "C:\Users\tayya\PycharmProjects\untitled\venv\lib\site-packages\torch\nn\_reduction.py", line 36, in legacy_get_string
    if size_average and reduce:
RuntimeError: bool value of Tensor with more than one value is ambiguous

Any help would be greatly appreciated. I'm new to NN's so if i made a glaring mistake, my apologies.

Output:
A couple of things:
1. From your data I concluded this is a classification problem, and not a regression problem. Thus the use of MSELoss isn't very optimal, I changed this to BCELoss which should be more appropriate.
2. The last activation of your network is RELU, and because this is a binary classification problem, Sigmoid is a better choice.
3. made a small fix in the "get_num_correct" function.

Hope this works for you:

from torch.utils.data import Dataset, DataLoader
import torch.nn as nn
from torch.nn import functional as F
from DataSetLoader import ReplaysDataSet
import torch

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.l1 = torch.nn.Linear(12,24)
        self.l2 = torch.nn.Linear(24, 20)
        self.l3 = torch.nn.Linear(20, 16)
        self.l4 = torch.nn.Linear(16, 10)
        self.l5 = torch.nn.Linear(10, 6)
        self.l6 = torch.nn.Linear(6, 1)

    def forward(self, t):
        t = F.relu(self.l1(t))
        t = F.relu(self.l2(t))
        t = F.relu(self.l3(t))
        t = F.relu(self.l4(t))
        t = F.relu(self.l5(t))
        y_pred = F.sigmoid(self.l6(t))

        return y_pred
    def get_num_correct(self,preds,labels):
        return preds.round().squeeze().eq(labels).numpy().sum()

class Test():
    model = Model()
    optimiser = torch.optim.SGD(model.parameters(), lr=0.01)

    dataset = ReplaysDataSet()
    trainLoader = DataLoader(dataset=dataset, batch_size=250, shuffle=True)
    batch = next(iter(trainLoader))
    criterion = nn.BCELoss()
    for epoch in range(10):
        totalLoss = 0
        totalCorrect = 0
        for batch in trainLoader:

            data, label = batch
            prediction = model(data)
            print(prediction)
            print(label)
            loss = criterion(prediction.squeeze(), label)
            optimiser.zero_grad()

            loss.backward()
            optimiser.step()

            totalLoss += loss.item()
            print(totalLoss)
            totalCorrect += model.get_num_correct(prediction, label)
            print(totalCorrect)


Instruction: Slicing a 4D tensor with a 3D tensor-index in PyTorch

Input:
I have a 4D tensor (which happens to be a stack of three batches of 56x56 images where each batch has 16 images) with the size of [16, 3, 56, 56]. My goal is to select the correct one of those three batches (with my index map that has the size of [16, 56, 56]) for each pixel and get the images that I want.

Now, I want to select the particular batches of images inside those three batches, with a  which has values such as

       [[[ 0,  0,  2,  ...,  0,  0,  0],
         [ 0,  0,  2,  ...,  0,  0,  0],
         [ 0,  0,  0,  ...,  0,  0,  0],
         ...,
         [ 0,  0,  0,  ...,  0,  0,  0],
         [ 0,  2,  0,  ...,  0,  0,  0],
         [ 0,  2,  2,  ...,  0,  0,  0]],

        [[ 0,  2,  0,  ...,  1,  1,  0],
         [ 0,  2,  0,  ...,  0,  0,  0],
         [ 0,  0,  0,  ...,  0,  2,  0],
         ...,
         [ 0,  0,  0,  ...,  0,  2,  0],
         [ 0,  0,  2,  ...,  0,  2,  0],
         [ 0,  0,  2,  ...,  0,  0,  0]]]

So for the 0s, the value will be selected from the first batch, where 1 and 2 will mean I want to select the values from the second and the third batch.

Here are some of the visualizations of the indices, each color denoting another batch.

I have tried to transpose the 4D tensor to match the dimensions of my indices, but it did not work. All it does is to give me a copy of the dimensions I have tried to select. Means

tposed = torch.transpose(fourD, 0,1) print(indices.size(),
outs.size(), tposed[:, indices].size())

outputs

torch.Size([16, 56, 56]) torch.Size([16, 3, 56, 56]) torch.Size([3, 16, 56, 56, 56, 56])

while the shape I need is

torch.Size([16, 56, 56]) or torch.Size([16, 1, 56, 56])

and as an example, if I try to select the right values for only the first image on the batch with

fourD[0,indices].size()

I get a shape like

torch.Size([16, 56, 56, 56, 56])

Not to mention that I get an out of memory error when I try this on the whole tensor.

I appreciate any help for using these indices to select either one of these three batches for each pixel in my images.

Note :

I have tried the option

outs[indices[:,None,:,:]].size()

and that returns

torch.Size([16, 1, 56, 56, 3, 56, 56])

Edit : torch.take does not help much since it treats the input tensor as a single dimensional array.

Output:
Turns out there is a function in PyTorch that has the functionality I was searching for.

torch.gather(fourD, 1, indices.unsqueeze(1))

did the job.

Here is a beautiful explanation of what gather does.


Instruction: Sampling point pairs from a grid in Pytorch

Input:
I need sample point pairs from a grid in PyTorch.

I have a tensor of size (1 x 500 x 1000). I also have a mask tensor of size(1 x 500 x 1000), denoting if a point is valid or not. I want to sample 200k point pairs from this grid. In other words, I want to get coordinates of sampled point pairs as a tensor of size (200k x 4), denoting (x1, y1, x2, y2) for all 200k point pairs. All points in the pairs should be valid points.

This will be repeated many times, so I need to have an efficient way of performing this procedure. What is an elegant way to implement this in PyTorch?

Output:
Not an expert here, but I did spend some time trying things out.
Turns out operating on 1D array is a lot faster (method two).

import time
import torch
class Timer():
    def __init__(self):
        pass
    def __enter__(self):
        self.time = time.time()
    def __exit__(self, *exc):
        print(f'time used: {time.time() - self.time:.2f}s')

# a = torch.rand([1,500,1000])
m = torch.randint(2, [1, 500, 1000]) # mask tensor
valid_len = (m==1).nonzero().size()[0] # number of valid points
rand_one = torch.randint(valid_len, [200000]) # sample 200k of random int
rand_two = torch.randint(valid_len, [200000]) # sample 200k of random int

# method one
m0 = m == 1 # mask of shape torch.Size([1, 500, 1000])
m0 = m0.nonzero() # valid points of shape torch.Size([valid_len, 3])
m0 = m0[:, 1:] # reshape to shape torch.Size([valid_len, 2])
with Timer():
    one0 = torch.index_select(m0, 0, rand_one) # take 200k valid points
    two0 = torch.index_select(m0, 0, rand_two) # take 200k valid points again
    coor0 = torch.cat([one0, two0], dim=1) # stack them up
# >>> time used: 1.05s

# method two
m1 = m.reshape(-1) # reshape mask to torch.Size([500000])
m1 = m1==1 # mask of shape torch.Size([500000])
m1 = m1.nonzero() # valid points of shape torch.Size([valid_len, 1])
m1 = m1.reshape(-1) # valid points of shape torch.Size([valid_len])
with Timer():
    one1 = m1.take(rand_one) # take 200k valid points
    two1 = m1.take(rand_two) # again
    # transform them to coordinates and stack them up
    coor1 = torch.stack([one1 // 1000, one1 % 1000, two1 // 1000, two1 % 1000], dim=1)
# >>> time used: 0.07s

assert torch.sum(coor0 == coor1) == 800000 # make sure consistent result

cheers


Instruction: How to implement some trainable parameters in the model of Keras like nn.Parameters() in Pytorch?

Input:
I just wanna to implement some trainable parameters in my model with Keras. In Pytorch, we can do it by using torch.nn.Parameter() like below:

self.a = nn.Parameter(torch.ones(8))
self.b = nn.Parameter(torch.zeros(16,8))

I think by doing this in pytorch it can add some trainable parameters into the model. And now I wanna to know, how to achieve similar operations in keras？
Any suggestions or advice are welcomed!

THX! :)

p.s. I just write a custom layer in Keras as below:

class Mylayer(Layer):

    def __init__(self,input_dim,output_dim,**kwargs):
        self.input_dim = input_dim
        self.output_dim = output_dim
        super(Mylayer,self).__init__(**kwargs)

    def build(self):

        self.kernel = self.add_weight(name='pi',
                                      shape=(self.input_dim,self.output_dim),
                                      initializer='zeros',
                                      trainable=True)
        self.kernel_2 = self.add_weight(name='mean',
                                        shape=(self.input_dim,self.output_dim),
                                        initializer='ones',
                                        trainable=True)

        super(Mylayer,self).build()

    def call(self,x):
        return x,self.kernel,self.kernel_2

and I wanna to know if I haven't change the tensor which pass through the layer, should I write the function def compute_output_shape() for necessary?

Output:
You need to create the trainable weights in a custom layer:

class MyLayer(Layer):
    def __init__(self, my_args, **kwargs):
        #do whatever you need with my_args

        super(MyLayer, self).__init__(**kwargs)

    #you create the weights in build:
    def build(self, input_shape):
        #use the input_shape to infer the necessary shapes for weights
        #use self.whatever_you_registered_in_init to help you, like units, etc.

        self.kernel = self.add_weight(name='kernel',
                                  shape=the_shape_you_calculated,
                                  initializer='uniform',
                                  trainable=True)

        #create as many weights as necessary for this layer

        #build the layer - equivalent to self.built=True
        super(MyLayer, self).build(input_shape)

    #create the layer operation here
    def call(self, inputs):
        #do whatever operations are needed
        #example:
        return inputs * self.kernel #make sure the shapes are compatible

    #tell keras about the output shape of your layer
    def compute_output_shape(self, input_shape):
        #calculate the output shape based on the input shape and your layer's rules
        return calculated_output_shape

Now use your layer in the model.

If you are using eager execution on with tensorflow and creating a custom training loop, you can work pretty much the same way you do with PyTorch, and you can create weights outside layers with tf.Variable, passing them as parameters to the gradient calculation methods.


Instruction: PIL Image.open display image reversely rotated

Input:
I'm working on predicting the number picture as  below with MNIST dataset and LeNet Model . Firstly, I show test images with Image.open, it displays test images in the way of reversely rotated.

from PIL import Image
import matplotlib.cm as cm
import pylab as pl
import numpy as np

img = Image.open('./test/2.png').convert('L')
img = np.invert(img) # convert to white on black
pl.imshow(np.asarray(img), origin='lower', cmap=cm.Greys_r)
pl.show()

Another issue is  the accuracy ratio of prediction seem very low. For example, the 2 here has been predicted as 4.

Someone could help with that or explain?  In my point of views, this number is much clear than handwriting MNIST. Thanks a lot.

Output:
imshow just sees an array of data. So specifying origin='lower' means you're telling imshow that the origin of your data is in the lower corner. However, image data has its origin in the upper corner so you can either remove origin= completely (the default is 'upper') or specify 'upper'.

pl.imshow(np.asarray(img), cmap=cm.Greys_r)

or

pl.imshow(np.asarray(img), origin='upper', cmap=cm.Greys_r)


Instruction: copy construct from a tensor: USER WARNING

Input:
I am creating a random tensor from normal distribution and since this tensor is served as the weight in the NN, to add requires_grad attributes, I use torch.tensor() as below:

import torch

input_dim, hidden_dim = 3, 5

norm = torch.distributions.normal.Normal(loc=0, scale=0.01)
W = norm.sample((input_dim, hidden_dim))
W = torch.tensor(W, requires_grad=True)

I am getting user warning error as below:

    UserWarning: To copy construct from a tensor,
    it is recommended to use sourceTensor.clone().detach() or
sourceTensor.clone().detach().requires_grad_(True),
rather than torch.tensor(sourceTensor).

Is there an alternative way to achieve the above? Thanks

Output:
You can just set W.requires_grad to True

import torch

input_dim, hidden_dim = 3, 5

norm = torch.distributions.normal.Normal(loc=0, scale=0.01)
W = norm.sample((input_dim, hidden_dim))
W.requires_grad = True


Instruction: Transform images to white on black and predict

Input:
I want to predict new customized images with LeNet model trained from here.
The customized images are black on white, so I need to convert them to white on black.

# Load & transform image
ori_img = Image.open('./test/2.png').convert('L')
img = np.invert(ori_img) #Transform images to white on black
t = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])
img = torch.autograd.Variable(t(img).unsqueeze(0))
ori_img.close()

# Predict
model.eval()
output = model(img)
pred = output.data.max(1, keepdim=True)[1][0][0]
print('Prediction: {}'.format(pred))

The result I got:

TypeError                                 Traceback (most recent call last)
<ipython-input-182-abbffa2ce0d8> in <module>
      7     transforms.Normalize((0.1307,), (0.3081,))
      8 ])
----> 9 img = torch.autograd.Variable(t(img).unsqueeze(0))
     10 ori_img.close()

~/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py in __call__(self, img)
     59     def __call__(self, img):
     60         for t in self.transforms:
---> 61             img = t(img)
     62         return img
     63

~/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py in __call__(self, img)
    196             PIL Image: Rescaled image.
    197         """
--> 198         return F.resize(img, self.size, self.interpolation)
    199
    200     def __repr__(self):

~/.local/lib/python3.6/site-packages/torchvision/transforms/functional.py in resize(img, size, interpolation)
    236     """
    237     if not _is_pil_image(img):
--> 238         raise TypeError('img should be PIL Image. Got {}'.format(type(img)))
    239     if not (isinstance(size, int) or (isinstance(size, Iterable) and len(size) == 2)):
    240         raise TypeError('Got inappropriate size arg: {}'.format(size))

TypeError: img should be PIL Image. Got <class 'numpy.ndarray'>

When I comment img = np.invert(ori_img) I get no errors, but all prediction results are 2s.

Someone could help? Thanks  a lot.

Output: You can this function: PIL.Image.fromarray to create a PIL Image from your numpy array, and then you can use the PIL.ImageOps.invert function to invert colors. Then your img variable should be the right type and inverted.


Instruction: Pytorch RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0

Input:
I use code from here to train a model to predict printed style number from 0 to 9:

idx_to_class = {0: "0", 1: "1", 2: "2", 3: "3", 4: "4", 5: "5", 6: "6", 7:"7", 8: "8", 9:"9"}
def predict(model, test_image_name):

    transform = image_transforms['test']

    test_image = Image.open(test_image_name)
    plt.imshow(test_image)

    test_image_tensor = transform(test_image)

    if torch.cuda.is_available():
        test_image_tensor = test_image_tensor.view(1, 3, 224, 224).cuda()
    else:
        test_image_tensor = test_image_tensor.view(1, 3, 224, 224)

    with torch.no_grad():
        model.eval()
        # Model outputs log probabilities
        out = model(test_image_tensor)
        ps = torch.exp(out)
        topk, topclass = ps.topk(1, dim=1)
        # print(topclass.cpu().numpy()[0][0])
        print("Image class:  ", idx_to_class[topclass.cpu().numpy()[0][0]])

predict(model, "path_of_test_image")

But I get an error when try to use predict:

Traceback (most recent call last):

  File "<ipython-input-12-f8636d3ba083>", line 26, in <module>
    predict(model, "/home/x/文档/Deep_Learning/pytorch/MNIST/test/2/QQ截图20191022093955.png")

  File "<ipython-input-12-f8636d3ba083>", line 9, in predict
    test_image_tensor = transform(test_image)

  File "/home/x/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)

  File "/home/x/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 166, in __call__
    return F.normalize(tensor, self.mean, self.std, self.inplace)

  File "/home/x/.local/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 217, in normalize
    tensor.sub_(mean[:, None, None]).div_(std[:, None, None])

RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0

How could I fix it? Thanks.

Output:
I suspect your test_image has an additional alpha channel per pixel, thus it has 4 channels instead of only three.
Try:
test_image = Image.open(test_image_name).convert('RGB')


Instruction: Tensorflow 2.0 dataset and dataloader

Input:
I am a pytorch user, and I am used to the data.dataset and data.dataloader api in pytorch. I am trying to build a same model with tensorflow 2.0, and I wonder whether there is an api that works similarly with these api in pytorch.

If there is no such api, can any of you tell me how people usually do to implement the data loading part in tensorflow ? I've used tensorflow 1, but never had an experience with dataset api. I've hard coded before. I hope there is something like overriding getitem with only index as an input.

Thanks much in advance.

Output:
When using the tf.data API, you will usually also make use of the map function.

In PyTorch, your __getItem__ call basically fetches an element from your data structure given in __init__ and transforms it if necessary.

In TF2.0, you do the same by initializing a Dataset using one of the Dataset.from_... functions (see from_generator, from_tensor_slices, from_tensors); this is essentially the __init__ part of a PyTorch Dataset. Then, you can call map to do the element-wise manipulations you would have in __getItem__.

Tensorflow datasets are pretty much fancy iterators, so by design you don't access their elements using indices, but rather by traversing them.

The guide on tf.data is very useful and provides a wide variety of examples.


Instruction: Pytorch Autograd: what does runtime error "grad can be implicitly created only for scalar outputs" mean

Input:
I am trying to understand Pytorch autograd in depth; I would like to observe the gradient of a simple tensor after going through a sigmoid function as below:

import torch
from torch import autograd

D = torch.arange(-8, 8, 0.1, requires_grad=True)

with autograd.set_grad_enabled(True):
    S = D.sigmoid()
S.backward()

My goal is to get D.grad() but even before calling it I get the runtime error:

RuntimeError: grad can be implicitly created only for scalar outputs

I see another post with similar question but the answer over there is not applied to my question.  Thanks

Output:
The error means you can only run .backward (with no arguments) on a unitary/scalar tensor. I.e. a tensor with a single element.

For example, you could do

T = torch.sum(S)
T.backward()

since T would be a scalar output.

I posted some more information on using pytorch to compute derivatives of tensors in this answer.