Instruction: Pre-Trained Models in Keras,TorchVision

Input:
I Have The Following Code which use pre-trained ResNet50 Model in Keras with imagenet DataSet:

from keras.applications.resnet50 import ResNet50
from keras.preprocessing import image
from keras.applications.resnet50 import preprocess_input, decode_predictions
import numpy as np

model = ResNet50(weights='imagenet')
print(model)

and it works fine..My question is can i find a pre-trained model inside Keras or Torchvision or TensorFlow But to one of the following:

1) LeNet5 for MNIST DataSet
2) 32-Layer ResNet for the CIFAR-10 Dataset

I know the alternative is to train the LeNet5 for example on my own, but a pre-trained model will be preferred and as far as I searched I didn't find them.
thank you.

Output:
I've also been exploring Tensorflow's pretrained model landscape and (as of 1/14/2020), solutions don't exist for 1) a mnist-pretrained lenet or 2) a cifar10-pretrained 32-layer resnet.

Honestly, I strongly doubt that most frameworks release a pretrained model for LeNet-5. It's extremely small and usually takes O(minutes) to train.

Aside from the tf.keras.applications module you mentioned, some other potential options are:

The official tensorflow/models repository, which contains model examples for mnist and a 32-layer resnet
Swithing to pytorch, which has various models pretrained on imagenet

I realize neither of these are ideal.


Instruction: How to do 2-layer nested FOR loop in PYTORCH?

Input:
I am learning to implement the Factorization Machine in Pytorch.
And there should be some feature crossing operations.
For example, I've got three features [A,B,C], after embedding, they are [vA,vB,vC], so the feature crossing is "[vA·vB], [vA·vC], [vB·vc]".

I know this operation can be simplified by the following:

It can be implemented by MATRIX OPERATIONS.
But this only gives a final result, say, a single value.

The question is, how to get all cross_vec in the following without doing FOR loop:
note: size of "feature_emb" is [batch_size x feature_len x embedding_size]

    g_feature = 0
    for i in range(self.featurn_len):
        for j in range(self.featurn_len):
            if j <= i: continue
            cross_vec = feature_emb[:,i,:] * feature_emb[:,j,:]
            g_feature += torch.sum(cross_vec, dim=1)

Output:
You can

cross_vec = (feature_emb[:, None, ...] * feature_emb[..., None, :]).sum(dim=-1)

This should give you corss_vec of shape (batch_size, feature_len, feature_len).

Alternatively, you can use torch.bmm

cross_vec = torch.bmm(feature_emb, feature_emb.transpose(1, 2))


Instruction: PyTorch conversion between tensor and numpy array: the addition operation

Input:
I am following the 60-minute blitz on PyTorch but have a question about conversion of a numpy array to a tensor. Tutorial example here.
This piece of code:
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)

yields

[2. 2. 2. 2. 2.]
tensor([2., 2., 2., 2., 2.], dtype=torch.float64)

However
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)
a = a + 1 #the diff is here
print(a)
print(b)

yields

[2. 2. 2. 2. 2.]
tensor([1., 1., 1., 1., 1.], dtype=torch.float64)

Why are the outputs different?

Output:
This actually has little to do with PyTorch. Compare

import numpy as np
a = np.ones(5)
b = a

followed by either

np.add(a, 1, out=a)
print(b)

or

a = a + 1
print(b)

There is a difference between np.add(a, 1, out=a) and a = a + 1. In the former you retain the same object (array) a with different values (2 instead of 1); in the latter you get a new array, which is bound to the same variable name a and has values of 2. However, the "original" a is discarded and unless something else (b) points to it, would be deallocated. In other words, the first operation is in-place and the latter out-of-place. Since b holds on to the array originally found at a, reassigning a + 1 to a does not affect the value of b. An alternative in-place mutation syntax would be

a[:] = a + 1
print(b)

Regarding PyTorch, it's very simple. from_numpy creates a tensor which aliases an actual object (array), so it is equivalent to the b = a line in my first snippet. The tensor will track the changes in the array named a at the point of calling, rather than the changes of what the name a points to.


Instruction: Problems passing tensor to linear layer - Pytorch

Input:
I'm trying to build a neural net however I can't figure out where I'm going wrong with the max pooling layer.

    self.embed1 = nn.Embedding(256, 8)
    self.conv_1 = nn.Conv2d(1, 64, (7,8), padding = (0,0))
    self.fc1 = nn.Linear(64, 2)

def forward(self,x):

    import pdb; pdb.set_trace()
    x = self.embed1(x) #input a tensor of ([1,217]) output size: ([1, 217, 8])
    x = x.unsqueeze(0) #conv lay needs a tensor of size (B x C x W x H) so unsqueeze here to make ([1, 1, 217, 8])
    x = self.conv_1(x) #creates 64 filter of size (7, 8).Outputs ([1, 64, 211, 1]) as 6 values lost due to not padding.

    x = torch.max(x,0) #returning max over the 64 columns. This returns a tuple of length 2 with 64 values in each att, the max val and indices.
    x = x[0] #I only need the max values. This returns a tensor of size ([64, 211, 1])
    x = x.squeeze(2) #linear layer only wants the number of inputs and number of outputs so I squeeze the tensor to ([64, 211])
    x = self.fc1(x) #Error Size mismatch (M1: [64 x 211] M2: [64 x 2])

I understand why the linear layer isn't accepting 211 however I don't understand why my tensor after maxing over the columns isn't 64 x 2.

Output:
You use of torch.max returns two outputs: the max value along dim=0 and the argmax along that dimension. Thus, you need to pick only the first output. (you might want to consider using adaptive max pooling for this task).

Your linear layer expects its input to have dim 64 (that is batch_size-by-64 shaped tensor). However, it seems like your x[0] is of shape 13504x1 - definitely not 64.

See this thread for example.


Instruction: How does Pytorch Dataloader handle variable size data?

Input:
I have a dataset that looks like below. That is the first item is the user id followed by the set of items which is clicked by the user.

0   24104   27359   6684
0   24104   27359
1   16742   31529   31485
1   16742   31529
2   6579    19316   13091   7181    6579    19316   13091
2   6579    19316   13091   7181    6579    19316
2   6579    19316   13091   7181    6579    19316   13091   6579
2   6579    19316   13091   7181    6579
4   19577   21608
4   19577   21608
4   19577   21608   18373
5   3541    9529
5   3541    9529
6   6832    19218   14144
6   6832    19218
7   9751    23424   25067   12606   26245   23083   12606

I define a custom dataset to handle my click log data.

import torch.utils.data as data
class ClickLogDataset(data.Dataset):
    def __init__(self, data_path):
        self.data_path = data_path
        self.uids = []
        self.streams = []

        with open(self.data_path, 'r') as fdata:
            for row in fdata:
                row = row.strip('\n').split('\t')
                self.uids.append(int(row[0]))
                self.streams.append(list(map(int, row[1:])))

    def __len__(self):
        return len(self.uids)

    def __getitem__(self, idx):
        uid, stream = self.uids[idx], self.streams[idx]
        return uid, stream

Then I use a DataLoader to retrieve mini batches from the data for training.

from torch.utils.data.dataloader import DataLoader
clicklog_dataset = ClickLogDataset(data_path)
clicklog_data_loader = DataLoader(dataset=clicklog_dataset, batch_size=16)

for uid_batch, stream_batch in stream_data_loader:
    print(uid_batch)
    print(stream_batch)

The code above returns differently from what I expected, I want stream_batch to be a 2D tensor of type integer of length 16. However, what I get is a list of 1D tensor of length 16, and the list has only one element, like below. Why is that ?

#stream_batch
[tensor([24104, 24104, 16742, 16742,  6579,  6579,  6579,  6579, 19577, 19577,
        19577,  3541,  3541,  6832,  6832,  9751])]

Output: So how do you handle the fact that your samples are of different length? torch.utils.data.DataLoader has a collate_fn parameter which is used to transform a list of samples into a batch. By default it does this to lists. You can write your own collate_fn, which for instance 0-pads the input, truncates it to some predefined length or applies any other operation of your choice.


Instruction: PyTorch model validation: The size of tensor a (32) must match the size of tensor b (13)

Input:
I am a very beginner in case of machine learning. So for learning purpose I am trying to develop a simple CNN to classify chess pieces. The net already works and I can train it but I have a problem with my validation function.

I can't compare my prediction with my target_data because my prediction is only a tensor of size 13 while target.data is [batch_size]x13. I can't figure out where my mistake is. The PyTorch examples are almost all using this function to compare the prediction with the target data.

It would be really great if anybody could help me out here.

You can lookup the rest of the code here: https://github.com/michaelwolz/ChessML/blob/master/train.ipynb

def validate(model, validation_data, criterion):
    model.eval()
    loss = 0
    correct = 0

    for i in range(len(validation_data)):
        data, target = validation_data[i][0], validation_data[i][1]
        target = torch.Tensor(target)

        if torch.cuda.is_available():
            data = data.cuda()
            target = target.cuda()

        out = model(data)

        loss += criterion(out, target).item()

        _, prediction = torch.max(out.data, 1)
        correct += (prediction == target.data).sum().item()

    loss = loss / len(validation_data)
    print("###################################")
    print("Average loss:", loss)
    print("Accuracy:", 100. * correct / len(validation_data))
    print("###################################")

Error:

<ipython-input-6-6b21e2bfb8a6> in validate(model, validation_data,

criterion)
     17
     18         _, prediction = torch.max(out.data, 1)
---> 19         correct += (prediction == target.data).sum().item()
     20
     21     loss = loss / len(validation_data)

RuntimeError: The size of tensor a (32) must match the size of tensor b (13) at non-singleton dimension 1

Edit:
My labels look like this:

[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Each index represents one class.
The output of the torch.max() function seems to be the index of the class.
I don't understand how I could compare the index to the target_label. I mean I could just write a function which checks if there is a 1 at the predicted index but I think that my mistake is somewhere else.

Output:
Simply run "argmax" on the target as well:

_, target = torch.max(target.data, 1)

Or better yet, just keep the target around as [example_1_class, example_2_class, ...], instead of 1-hot encoding.


Instruction: Unexpected key(s) in state_dict: "model", "opt"

Input:
I'm currently using fast.ai to train an image classifier model.

data = ImageDataBunch.single_from_classes(path, classes, ds_tfms=get_transforms(), size=224).normalize(imagenet_stats)
learner = cnn_learner(data, models.resnet34)

learner.model.load_state_dict(
    torch.load('stage-2.pth', map_location="cpu")
)

which results in :

  torch.load('stage-2.pth', map_location="cpu")   File
  "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py",
  line 769, in load_state_dict
      self.class.name, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for Sequential:

  ...

  Unexpected key(s) in state_dict: "model", "opt".

I have looked around in SO and tried to use the following solution:

# original saved file with DataParallel
state_dict = torch.load('stage-2.pth', map_location="cpu")
# create new OrderedDict that does not contain `module.`
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove `module.`
    new_state_dict[name] = v
# load params
learner.model.load_state_dict(new_state_dict)

which results in :

  RuntimeError: Error(s) in loading state_dict for Sequential:

  Unexpected key(s) in state_dict: "".

I'm using Google Colab to train my model and then port the trained model into docker and try to host in in a local server.

What could be the issue? Could it be the different version of pytorch which results in model mismatch?

In my docker config:

# Install pytorch and fastai
RUN pip install torch_nightly -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
RUN pip install fastai

While my Colab is using the following:

 !curl -s https://course.fast.ai/setup/colab | bash

Output:
My strong guess is that stage-2.pth contains two top-level items: the model itself (its weights) and the final state of the optimizer which was used to train it. To load just the model, you need only the former. Assuming things were done in the idiomatic PyTorch way, I would try

learner.model.load_state_dict(
    torch.load('stage-2.pth', map_location="cpu")['model']
)

Update: after applying my first round of advice it becomes clear that you're loading a savepoint create with a different (perhaps differently configured?) model than the one you're loading it into. As you can see in the pastebin, the savepoint contains weights for some extra layers, not present in your model, such as bn3, downsample, etc.

  "0.4.0.bn3.running_var", "0.4.0.bn3.num_batches_tracked", "0.4.0.downsample.0.weight"

at the same time some other key names match, but the tensors are of different shapes.

  size mismatch for 0.5.0.downsample.0.weight: copying a param with shape torch.Size([512, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([128, 64, 1, 1]).

I see a pattern that you consistently try to load a parameter of shape [2^(x+1), 2^x, 1, 1] in place of [2^(x), 2^(x-1), 1, 1]. Perhaps you're trying to load a model of different depth (ex. loading vgg-16 weights for vgg-11?). Either way, you need to figure out the exact architecture used to create your savepoint and then recreate it before loading the savepoint.

PS. In case you weren't sure - savepoints contain model weights, along with their shapes and (autogenerated) names. They do not contain the full specification of the architecture itself - you need to assure yourself, that you're calling model.load_state_dict with model being of exactly the same architecture as was used to create the savepoint. Otherwise you will likely have weight names mismatching.


Instruction: converting list of tensors to tensors pytorch

Input:
I have list of tensor where each tensor has a different size. How can I convert this list of tensors into a tensor using PyTorch?
For instance,
x[0].size() == torch.Size([4, 8])
x[1].size() == torch.Size([4, 7])  # different shapes!

This:
torch.tensor(x)

Gives the error:

ValueError: only one element tensors can be converted to Python scalars

Output:
You might be looking for cat.
However, tensors cannot hold variable length data.
for example, here we have a list with two tensors that have different sizes(in their last dim(dim=2)) and we want to create a larger tensor consisting of both of them, so we can use cat and create a larger tensor containing both of their data.
also note that you can't use cat with half tensors on cpu as of right now so you should convert them to float, do the concatenation and then convert back to half
import torch

a = torch.arange(8).reshape(2, 2, 2)
b = torch.arange(12).reshape(2, 2, 3)
my_list = [a, b]
my_tensor = torch.cat([a, b], dim=2)
print(my_tensor.shape) #torch.Size([2, 2, 5])

you haven't explained your goal so another option is to use pad_sequence like this:
from torch.nn.utils.rnn import pad_sequence
a = torch.ones(25, 300)
b = torch.ones(22, 300)
c = torch.ones(15, 300)
pad_sequence([a, b, c]).size() #torch.Size([25, 3, 300])

edit: in this particular case, you can use torch.cat([x.float() for x in sequence], dim=1).half()


Instruction: RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 3 and 1 in dimension 1

Input:
While training the resnet50 model through pytorch I got this error:

  RuntimeError: invalid argument 0: Sizes of tensors must match except
  in dimension 0. Got 3 and 1 in dimension 1 at
  /pytorch/aten/src/TH/generic/THTensorMoreMath.cpp:1333

I'm using this: http://github.com/Helias/Car-Model-Recognition/

with this dataset http://vmmrdb.cecsresearch.org/

Output:
I solved this, the problem was the different images color channels, not all the images were RGB, so I made a conversion in dataset.py, I changed this:

im = Image.open(image_path)

into this:

im = Image.open(image_path).convert('RGB')


Instruction: Pytorch batch matrix vector outer product

Input:
I am trying to generate a vector-matrix outer product (tensor) using PyTorch. Assuming the vector v has size p and the matrix M has size qXr, the result of the product should be pXqXr.

Example:

#size: 2
v = [0, 1]
#size: 2X3
M = [[0, 1, 2],
     [3, 4, 5]]
#size: 2X2X3
v*M = [[[0, 0, 0],
        [0, 0, 0]],
       [[0, 1, 2],
        [3, 4, 5]]]

For two vectors v1 and v2, I can use torch.bmm(v1.view(1, -1, 1), v2.view(1, 1, -1)). This can be easily extended for a batch of vectors. However, I am not able to find a solution for vector-matrix case. Also, I need to do this operation for batches of vectors and matrices.

Output:
You can use torch.einsum operator:
torch.einsum('bp,bqr->bpqr', v, M) # batch-wise operation v.shape=(b,p) M.shape=(b,q,r)
torch.einsum('p,qr->pqr', v, M)    # cross-batch operation


Instruction: PyTorch PermissionError: [Errno 13] Permission denied: '/.torch'

Input:
I'm running a PyTorch based ML program for image classification using Resnet50 model for transfer learning. I am getting below error regarding permission.

Traceback (most recent call last):
File "imgc_pytorch.py", line 67, in
  model = models.resnet50(pretrained=True)
File "/opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py", line 187, in resnet50
  model.load_state_dict(model_zoo.load_url(model_urls['resnet50']))
File "/opt/conda/lib/python3.6/site-packages/torch/utils/model_zoo.py",
line 59, in load_url
  os.makedirs(model_dir)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
  makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
  mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.torch'

Looked up on this forum and it was suggested to add 'sudo' before the file name, but getting error "/bin/sh: 1: sudo: not found"

Output: you can change model_zoo.load_url(model_urls['resnet50']) to model_zoo.load_url(model_urls['resnet50'], model_dir='~/.torch/') like this


Instruction: Extract features from last hidden layer Pytorch Resnet18

Input:
I am implementing an image classifier using the Oxford Pet dataset with the pre-trained Resnet18 CNN.
The dataset consists of 37 categories with ~200 images in each of them.
Rather than using the final fc layer of the CNN as output to make predictions I want to use the CNN as a feature extractor to classify the pets.
For each image i'd like to grab features from the last hidden layer (which should be before the 1000-dimensional output layer). My model is using Relu activation so I should grab the output just after the ReLU (so all values will be non-negative)
Here is code (following the transfer learning tutorial on Pytorch):
loading data
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                std=[0.229, 0.224, 0.225])

image_datasets = {"train": datasets.ImageFolder('images_new/train', transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        normalize
    ])), "test": datasets.ImageFolder('images_new/test', transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        normalize
    ]))
               }

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4, pin_memory=True)
              for x in ['train', 'test']}

dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'test']}

train_class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

train function
def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'test']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):

                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)

                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'test' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

Compute SGD cross-entropy loss
model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features

print("number of features: ", num_ftrs)

model_ft.fc = nn.Linear(num_ftrs, len(train_class_names))

model_ft = model_ft.to(device)
criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=24)

Now how do I get a feature vector from the last hidden layer for each of my images? I know I have to freeze the previous layer so that gradient isn't computed on them but I'm having trouble extracting the feature vectors.
My ultimate goal is to use those feature vectors to train a linear classifier such as Ridge or something like that.
Thanks!

Output:
This is probably not the best idea, but you can do something like this:
#assuming model_ft is trained now
model_ft.fc_backup = model_ft.fc
model_ft.fc = nn.Sequential() #empty sequential layer does nothing (pass-through)
# or model_ft.fc = nn.Identity()
# now you use your network as a feature extractor

I also checked fc is the right attribute to change, look at forward


Instruction: How to save and load random number generator state in Pytorch?

Input:
I am training a DL model in Pytorch, and want to train my model in a deterministic way.
As written in this official guide, I set random seeds like this:

np.random.seed(0)
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Now, my training is long and i want to save, then later load everything, including the RNGs. I use torch.save and torch.load_state_dict for the model and the optimizer.

How can the random number generators be saved & loaded?

Output:
You can use torch.get_rng_state and  torch.set_rng_state

When calling torch.get_rng_state you will get your random number generator state as a torch.ByteTensor.

You can then save this tensor somewhere in a file and later you can load and use torch.set_rng_state to set the random number generator state.

When using numpy you can of course do the same there using:
numpy.random.get_state and numpy.random.set_state


Instruction: Build Pytorch from source

Input:
I'm trying to install Pytorch from source on my MacOS (version 10.14.3) to use GPU. I have follow the documentation from this link. When I launch in my terminal the MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install I'm getting the following error in my terminal:

[ 69%] Built target caffe2_observers make: *** [all] Error 2 Traceback (most recent
call last): File "setup.py", line 710, in <module>
build_deps()
File "setup.py", line 282, in build_deps
build_dir='build')
File "/Users/Desktop/pytorch/tools/build_pytorch_libs.py", line 259, in build_caffe2
check_call(['make', '-j', str(max_jobs), 'install'], cwd=build_dir, env=my_env)
File "/Users/anaconda3/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j', '4', 'install']'
returned non-zero exit status 2.

I tried to upgrade pip and reinstall anaconda and cuda without success.

Here you can find the error just belong :

[ 69%] Building CXX object modules/observers/CMakeFiles/caffe2_observers.dir/net_observer_reporter_print.cc.o
In file included from <built-in>:1:
In file included from /usr/local/cuda/include/cuda_runtime.h:115:
In file included from /usr/local/cuda/include/crt/common_functions.h:77:
/Library/Developer/CommandLineTools/usr/include/c++/v1/string.h:61:15: fatal error: 'string.h' file not found
#include_next <string.h>
          ^~~~~~~~~~
1 error generated.
CMake Error at caffe2_gpu_generated_THCReduceApplyUtils.cu.o.Release.cmake:219 (message):
    Error generating
/Users/Desktop/pytorch/build/caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/./caffe2_gpu_generated_THCReduceApplyUtils.cu.o

make[2]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/caffe2_gpu_generated_THCReduceApplyUtils.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
Scanning dependencies of target torch_shm_manager
1 error generated.

Output: I was encountering this problem because Nvidia is incompatible with OSX Mojave 10.14+


Instruction: Finding non-intersection of two pytorch tensors

Input:
Thanks everyone in advance for your help! What I'm trying to do in PyTorch is something like numpy's setdiff1d. For example given the below two tensors:

t1 = torch.tensor([1, 9, 12, 5, 24]).to('cuda:0')
t2 = torch.tensor([1, 24]).to('cuda:0')

The expected output should be (sorted or unsorted):

torch.tensor([9, 12, 5])

Ideally the operations are done on GPU and no back and forth between GPU and CPU. Much appreciated!

Output:
if you don't want to leave cuda, a workaround could be:

t1 = torch.tensor([1, 9, 12, 5, 24], device = 'cuda')
t2 = torch.tensor([1, 24], device = 'cuda')
indices = torch.ones_like(t1, dtype = torch.uint8, device = 'cuda')
for elem in t2:
    indices = indices & (t1 != elem)
intersection = t1[indices]


Instruction: Fastest way to read an image from huge uncompressed tar file in __getitem__ of PyTorch custom dataset

Input:
I have a huge dataset (2 million) of jpg images in one uncompressed TAR file. I also have a txt file each line is the name of the image in TAR file in order.

img_0000001.jpg
img_0000002.jpg
img_0000003.jpg
...

and images in tar file are exactly the same.
I searched alot and find out tarfile module is the best one, but when I tried to read images from tar file using name, it takes too long. And the reason is, everytime I call getmemeber(name) method, it calls getmembers() method which scan whole tar file, then return a Namespace of all names, then start finding in this Namespace.

if it helps, my dataset size is 20GB single tar file.

I don't know it is better idea to first extract all then use extracted folders in my CustomDataset or reading directly from archive.

Here is the code I am using to read a single file from tar file:

        with tarfile.open('data.tar') as tf:
            tarinfo = tf.getmember('img_000001.jpg')
            image = tf.extractfile(tarinfo)
            image = image.read()
            image = Image.open(io.BytesIO(image))

I used this code in my __getitem__ method of CustomDataset class that loops over all names in filelist.txt

Thanks for any advice

Output:
tarfile seems to have caching for getmember, it reuses getmembers() results.

But if you use the provided snipped in __getitem__, then for each item from the dataset the tar file is open and read fully, one image file extracted, then the tar file closed and the associated info is lost.

The simplest way to resolve this is probably to open the tar file in your dataset's __init__ like self.tf = tarfile.open('data.tar'), but then you need to remember to close it in the end.


Instruction: Convolutional auto-encoder error - 'RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.FloatTensor) should be the same'

Input:
For below model I received error 'Expected stride to be a single value integer or list'. I used suggested answer from https://discuss.pytorch.org/t/expected-stride-to-be-a-single-integer-value-or-a-list/17612/2
 and added

img.unsqueeze_(0)

I now receive error :

RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.FloatTensor) should be the same

For below code I three sample images and attempt to learn a representation of them using an auto-encoder :

%reset -f

import torch.utils.data as data_utils
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from matplotlib import pyplot as plt
from sklearn import metrics
import datetime
from sklearn.preprocessing import MultiLabelBinarizer
import seaborn as sns
sns.set_style("darkgrid")
from ast import literal_eval
import numpy as np
from sklearn.preprocessing import scale
import seaborn as sns
sns.set_style("darkgrid")
import torch
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable
from os import listdir
import cv2
import torch.nn.functional as F
import numpy as np
from numpy.polynomial.polynomial import polyfit
import matplotlib.pyplot as plt

number_channels = 3

%matplotlib inline

x = np.arange(10)
m = 1
b = 2
y = x * x
plt.plot(x, y)
plt.axis('off')
plt.savefig('1-increasing.jpg')

x = np.arange(10)
m = 0.01
b = 2
y = x * x * x
plt.plot(x, y)
plt.axis('off')
plt.savefig('2-increasing.jpg')

x = np.arange(10)
m = 0
b = 2
y = (m*x)+b
plt.plot(x, y)
plt.axis('off')
plt.savefig('constant.jpg')

batch_size_value = 2

train_image = []

train_image.append(cv2.imread('1-increasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('2-increasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('decreasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('constant.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))

data_loader = data_utils.DataLoader(train_image, batch_size=batch_size_value, shuffle=False,drop_last=True)

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.utils import save_image
from torchvision.datasets import MNIST
import os

if not os.path.exists('./dc_img'):
    os.mkdir('./dc_img')

def to_img(x):
    x = 0.5 * (x + 1)
    x = x.clamp(0, 1)
    x = x.view(x.size(0), 1, 28, 28)
    return x

num_epochs = 100
# batch_size = 128
batch_size = 2

learning_rate = 1e-3
dataloader = data_loader

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(3, 16, 3, stride=3, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2),  # b, 16, 5, 5
            nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
            nn.ReLU(True),
            nn.MaxPool3d(3, stride=1)  # b, 8, 2, 2
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose3d(8, 16, 3, stride=2),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose3d(16, 8, 5, stride=3, padding=1),  # b, 8, 15, 15
            nn.ReLU(True),
            nn.ConvTranspose3d(8, 1, 2, stride=2, padding=1),  # b, 1, 28, 28
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = autoencoder()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,
                             weight_decay=1e-5)

for epoch in range(num_epochs):
    for data in dataloader:
        img, _ = data
        img.unsqueeze_(0)
#         img.unsqueeze_(0)
#         print(img)
#         img.unsqueeze_(0)
        img = Variable(img).cuda()
        # ===================forward=====================
        output = model(img)
        loss = criterion(output, img)
        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    # ===================log=================to_img=======
    print('epoch [{}/{}], loss:{:.4f}'
          .format(epoch+1, num_epochs, loss.data[0]))
    if epoch % 10 == 0:
        pic = to_img(output.cpu().data)
        save_image(pic, './dc_img/image_{}.png'.format(epoch))

torch.save(model.state_dict(), './conv_autoencoder.pth')

But as stated earlier this results in error :

299     def forward(self, input):
300         return F.conv2d(input, self.weight, self.bias, self.stride,

  --> 301                         self.padding, self.dilation, self.groups)
      302
      303

  RuntimeError: Input type (torch.cuda.ByteTensor) and weight type
  (torch.FloatTensor) should be the same

The issue appears to be related to img.unsqueeze_(0) ?

How to train the auto-encoder on these images ?

Output:
This is because your image tensor resides in GPU (that happens here img = Variable(img).cuda()), while your model is still in RAM. Please remember that you need to explicitly call cuda() to send a tensor (or an instance of nn.Module) to GPU.

Just change this line:

model = autoencoder()

To this:

model = autoencoder().cuda()


Instruction: Groups in Convolutional Neural Network / CNN

Input:
I came across this PyTorch example for depthwise separable convolutions using the groups parameter:

class depthwise_separable_conv(nn.Module):
    def __init__(self, nin, nout):
        super(depthwise_separable_conv, self).__init__()
        self.depthwise = nn.Conv2d(nin, nin, kernel_size=3, padding=1, groups=nin)
        self.pointwise = nn.Conv2d(nin, nout, kernel_size=1)

    def forward(self, x):
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out

I haven't seen any usage of groups in CNNs before. The documentation is also a bit sparse as far as that is concerned:

  groups controls the connections between inputs and outputs.
  in_channels and out_channels must both be divisible by groups.

So my questions are:

What are groups in CNNs?
In which cases do I need to use groups?

(I guess this is more a general, not PyTorch specific.)

Output:
Perhaps you're looking up an older version of the docs. 1.0.1 documentation for nn.Conv2d expands on this.

Groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,
At groups=1, all inputs are convolved to all outputs.
At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.
At groups= in_channels, each input channel is convolved with its own set of filters, of size: (floor(c_out / c_in))

If you prefer a more mathematical description, start by thinking of a 1x1 convolution with groups=1 (default). It is a essentially a full matrix applied across all channels f at each (h, w) location. Setting groups to higher values turns this matrix into a diagonal block-sparse matrix with the number of blocks equal to groups. With groups=in_channels you get a diagonal matrix.
Now, if the kernel is larger than 1x1, you retain the channel-wise block-sparsity as above, but allow for larger spatial kernels. I suggest rereading the groups=2 exempt from the docs I quoted above, it describes exactly that scenario in yet another way, perhaps helpful for understanding. Hope this helps.
Edit: Why does anybody want to use it? Either as a constraint (prior) for the model or as a performance improvement technique; sometimes both. In the linked thread the idea is to replace a NxN, groups=1 2d conv with a sequence of NxN, groups=n_features -> 1x1, groups=1 convolutions. This mathematically results in a single convolution (since a convolution of a convolution is still a convolution), but makes the "product" convolution matrix more sparse and thus reduces the number of parameters and computational complexity. This seems to be a reasonable resource explaining this more in-depth.


Instruction: Output and Broadcast shape mismatch in MNIST, torchvision

Input:
I am getting following error when using MNIST dataset in Torchvision

RuntimeError: output with shape [1, 28, 28] doesn't match the broadcast shape [3, 28, 28]

Here is my code:

import torch
from torchvision import datasets, transforms

transform = transforms.Compose([transforms.ToTensor(),
                            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                          ])
trainset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
images, labels = next(iter(trainloader))

Output:
The error is due to color vs grayscale on the dataset, the dataset is grayscale.

I fixed it by changing transform to

transform = transforms.Compose([transforms.ToTensor(),
  transforms.Normalize((0.5,), (0.5,))
])


Instruction: Implementing fast dense feature extraction in PyTorch

Input:
I am trying to implement this paper in PyTorch Fast Dense Feature Extractor but I am having trouble converting the Torch implementation example they provide into PyTorch.

My attempt thus far has the issue that when adding an additional dimension to the feature map then the convolutional weights don't match the feature shape. How is this managed in Torch (from their implementation it seem that Torch doesn't care about this, but PyTorch does). My code: https://gist.github.com/system123/c4b8ef3824f2230f181f8cfba84f0cfd

Any other solutions to this problem would be great too. Basically, I have a feature extractor that converts a 128x128 patch into an embedding and I'd like to apply this in a dense manner across a larger image without using a for loop to evaluate the CNN on each location as that has a lot of duplicate computation.

Output:
It is your lucky day as I have recently uploaded a PyTorch and TF implementation of the paper Fast Dense Feature Extraction with CNNs with Pooling Layers.

An approach to compute patch-based local feature descriptors efficiently in presence of pooling and striding layers for whole images at once.

See https://github.com/erezposner/Fast_Dense_Feature_Extraction for details.

It contains simple instructions that will explain how to use the Fast Dense Feature Extraction (FDFE) project.

Good luck


Instruction: How can I make a neural network that has multiple outputs using pytorch?

Input: Is my question even right? I looked everywhere but couldn't find a single thing. I'm pretty sure this was addressed when I learned keras, but how do I implement it in pytorch?

Output:
Multiple outputs can be trivially achieved with pytorch.

Here is one such network.

import torch.nn as nn

class NeuralNetwork(nn.Module):
  def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.linear1 = nn.Linear(in_features = 3, out_features = 1)
    self.linear2 = nn.Linear(in_features = 3,out_features = 2)

  def forward(self, x):
    output1 = self.linear1(x)
    output2 = self.linear2(x)
    return output1, output2


Instruction: PyTorch grid_sample returns zero array

Input:
I want to sample rectangular patch from my image by affine_grid/grid_sample
I created array which contains only 255 values

canvas1 = np.zeros((128, 128), dtype=np.uint8)
canvas1[:] = 255

Also i created grid

theta = torch.FloatTensor([[
    [11/2, 0, 63],
    [0, 11/2, 63],
]])
grid = F.affine_grid(theta, (1, 1, 11, 11))

Grid contains values like

     [[57.5000, 57.5000],
      [58.6000, 57.5000],
      [59.7000, 57.5000],
      [60.8000, 57.5000],
      [61.9000, 57.5000],
      [63.0000, 57.5000],
      [64.1000, 57.5000],
      [65.2000, 57.5000],
      [66.3000, 57.5000],
      [67.4000, 57.5000],
      [68.5000, 57.5000]],
...............

After that i called grid_sample

canvas1_torch = torch.FloatTensor(canvas1.astype(np.float32))
canvas1_torch = canvas1_torch.unsqueeze(0).unsqueeze(0)
sampled = F.grid_sample(canvas1_torch, grid, mode="bilinear")

Unfortunately sampled contains zero values (but canvas1_torch[0, 0, 63, 65]) is 255

What i am doing wrong?

Output:
Your grid values are outside [-1, 1].

According to https://pytorch.org/docs/stable/nn.html#torch.nn.functional.grid_sample, such values are handled as defined by padding_mode.

Default padding_mode is 'zeros', what you probably want is "border": F.grid_sample(canvas1_torch, grid, mode="bilinear", padding_mode="border") returns all values 255.


Instruction: How to transfer the following tensorflow code into pytorch

Input:
I want to re-implement the word embedding here

here is the original tensorflow code  (version: 0.12.1)

import tensorflow as tf

class Network(object):
    def __init__(
        self, user_length,item_length, num_classes, user_vocab_size,item_vocab_size,fm_k,n_latent,user_num,item_num,
        embedding_size, filter_sizes, num_filters, l2_reg_lambda=0.0,l2_reg_V=0.0):

 # Skip the embedding
    pooled_outputs_u = []

    for i, filter_size in enumerate(filter_sizes):
        with tf.name_scope("user_conv-maxpool-%s" % filter_size):
            # Convolution Layer
            filter_shape = [filter_size, embedding_size, 1, num_filters]
            W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
            b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
            conv = tf.nn.conv2d(
                self.embedded_users,
                W,
                strides=[1, 1, 1, 1],
                padding="VALID",
                name="conv")
            # Apply nonlinearity
            h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
            # Maxpooling over the outputs
            pooled = tf.nn.max_pool(
                h,
                ksize=[1, user_length - filter_size + 1, 1, 1],
                strides=[1, 1, 1, 1],
                padding='VALID',
                name="pool")
            pooled_outputs_u.append(pooled)
    num_filters_total = num_filters * len(filter_sizes)
    self.h_pool_u = tf.concat(3,pooled_outputs_u)
    self.h_pool_flat_u = tf.reshape(self.h_pool_u, [-1, num_filters_total])

    with tf.name_scope("dropout"):
        self.h_drop_u = tf.nn.dropout(self.h_pool_flat_u, 1.0)
        self.h_drop_i= tf.nn.dropout(self.h_pool_flat_i, 1.0)
    with tf.name_scope("get_fea"):
        Wu = tf.get_variable(
            "Wu",
            shape=[num_filters_total, n_latent],
            initializer=tf.contrib.layers.xavier_initializer())
        bu = tf.Variable(tf.constant(0.1, shape=[n_latent]), name="bu")
        self.u_fea=tf.matmul(self.h_drop_u, Wu) + bu
        #self.u_fea = tf.nn.dropout(self.u_fea,self.dropout_keep_prob)
        Wi = tf.get_variable(
            "Wi",
            shape=[num_filters_total, n_latent],
            initializer=tf.contrib.layers.xavier_initializer())
        bi = tf.Variable(tf.constant(0.1, shape=[n_latent]), name="bi")
        self.i_fea = tf.matmul(self.h_drop_i, Wi) + bi
        #self.i_fea=tf.nn.dropout(self.i_fea,self.dropout_keep_prob)

    with tf.name_scope('fm'):
        self.z=tf.nn.relu(tf.concat(1,[self.u_fea,self.i_fea]))

        #self.z=tf.nn.dropout(self.z,self.dropout_keep_prob)

        WF1=tf.Variable(
                tf.random_uniform([n_latent*2, 1], -0.1, 0.1), name='fm1')
        Wf2=tf.Variable(
            tf.random_uniform([n_latent*2, fm_k], -0.1, 0.1), name='fm2')
        one=tf.matmul(self.z,WF1)

        inte1=tf.matmul(self.z,Wf2)
        inte2=tf.matmul(tf.square(self.z),tf.square(Wf2))

        inter=(tf.square(inte1)-inte2)*0.5

        inter=tf.nn.dropout(inter,self.dropout_keep_prob)

        inter=tf.reduce_sum(inter,1,keep_dims=True)
        print inter
        b=tf.Variable(tf.constant(0.1), name='bias')

And here is the pytorch version 1.0 that I try

class Network(torch.nn.Module):
    def __init__(
        self, user_length,item_length, num_classes, user_vocab_size,item_vocab_size,fm_k,n_latent,user_num,item_num,
        embedding_size, filter_sizes, num_filters, l2_reg_lambda=0.0,l2_reg_V=0.0):

        pooled_outputs_u = []

    def forward():

I mainly have the trouble with the convolutional layer tranforming.

Pytorch is much easy to use since we can skip lots of W and b definition. Is there some one can help me with the rest? Thanks

Output:
The pytorch equivalent of the tensorflow part of the code will be, explained with comments in the code itself, you have to import truncnorm from scipy.

from scipy.stats import truncnorm #extra import equivalent to tf.trunc initialise

pooled_outputs_u = []

for i, filter_size in enumerate(filter_sizes):

    filter_shape = [filter_size, embedding_size, 1, num_filters]

    #W is just a tensor now that will act as weight to the conv layer
    W=torch.tensor(truncnorm.rvs(-1,1,size=[10,10]))
    #bias initialized with 0.1 initial values
    b=torch.zeros([num_filters])+0.1
    #conv layer with the same parameters as the tensorflow layer more on this in the link
    conv=torch.nn.functional.conv2d(self.embedded_users,W,bias=b,strides=[1,1,1,1],padding=0)
    #can use torch sequential to include it all in a single line but did it like this for better understanding.
    h=torch.relu(conv)
    #look at link2 for what a max pool layer does, basically it is a kind of feature extraction
    pooled=torch.nn.functional.max_pool(h,kernal_size=[1,user_length-filter_size+1,1,1],strides=[1,1,1,1],padding=0)
    pooled_outputs_u.append(pooled)
num_filters_total = num_filters * len(filter_sizes)
self.h_pool_u=torch.concat(3,pooled_outputs_u)
self.h_pool_flat_u=torch.reshape(self.h_pool_u,[-1,num_filters_total])

Reference:

Link 1

Link 2


Instruction: How to transfer the follow Embedding code in tensorflow to pytorch?

Input:
I have an Embedding code in Tensorflow as follow

self.input_u = tf.placeholder(tf.int32, [None, user_length], name="input_u")
with tf.name_scope("user_embedding"):
        self.W1 = tf.Variable(
            tf.random_uniform([user_vocab_size, embedding_size], -1.0, 1.0),
            name="W")
        self.embedded_user = tf.nn.embedding_lookup(self.W1, self.input_u)
        self.embedded_users = tf.expand_dims(self.embedded_user, -1)

And I want to re-write in pytorch, How to do that?

Output:
Method 1: Use Embedding layer and freeze the weight to act as lookup table

import numpy as np
import torch
# user_vocab_size = 10
# embedding_size = 5
W1 = torch.FloatTensor(np.random.uniform(-1,1,size=(user_vocab_size,embedding_size)))
embedded_user = torch.nn.Embedding(user_vocab_size,embedding_size, _weight=W1)
embedded_user.weight.requires_grad = False
embedded_users = torch.unsqueeze(embedded_user, -1)

# user_length = 5
# batch_size = 4
#input = torch.LongTensor(np.random.randint(0,user_vocab_size,(batch_size,user_length)))
#embb = embedded_user(input)

You can change the dimensions of embb tensor to your needs using torch.unqueeze

W1 : A tensor of uniform distribution between (-1,1) of size (user_vocab_size,  embedding_size)
embedded_user : Is an embedding layer which uses W1 as embedding vectors

Method 2: Use Embedding functional api

input_u = torch.LongTensor(np.random.randint(0,user_vocab_size,(batch_size,user_length)))
embedded_user = torch.nn.functional.embedding(input_u,W1)
embedded_users = torch.unsqueeze(embedded_user, -1)


Instruction: Output from LSTM not changing for different inputs

Input:
I have the an LSTM implemented in PyTorch as below.

import numpy as np

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable

class LSTM(nn.Module):
    """
    Defines an LSTM.
    """

    def __init__(self, input_dim, hidden_dim, output_dim, num_layers):

        super(LSTM, self).__init__()

        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)

    def forward(self, input_data):

        lstm_out_pre, _ = self.lstm(input_data)
        return lstm_out_pre

model = LSTM(input_dim=2, hidden_dim=2, output_dim=1, num_layers=8)

random_data1 = torch.Tensor(np.random.standard_normal(size=(1, 5, 2)))
random_data2 = torch.Tensor(np.random.standard_normal(size=(1, 5, 2)))

out1 = model(random_data1).detach().numpy()
out2 = model(random_data2).detach().numpy()

print(out1)
print(out2)

I am simply creating an LSTM network and passing two random inputs into it. The outputs does not make sense because no matter what random_data1 and random_data2 is, out1 and out2 are always the same. This does not make any sense to me as random inputs multiplied with random weights should give different outputs.

This does not seem to be the case if I use less number of hidden layers. With num_layers=2, this effect seems to be nil. And as you increase it, out1 and out2 keeps on getting closer. This does not make sense to me because with more layers of the LSTM stacked on top of each other, we are multiplying the input with more number of random weights which should magnify the differences in the input and give a very different output.

Can someone please explain this behavior? Is there something wrong with my implementation?

In one particular run, random_data1 is

tensor([[[-2.1247, -0.1857],
         [ 0.0633, -0.1089],
         [-0.6460, -0.1079],
         [-0.2451,  0.9908],
         [ 0.4027,  0.3619]]])

random_data2 is

tensor([[[-0.9725,  1.2400],
         [-0.4309, -0.7264],
         [ 0.5053, -0.9404],
         [-0.6050,  0.9021],
         [ 1.4355,  0.5596]]])

out1 is

[[[0.12221643 0.11449362]
  [0.18342148 0.1620608 ]
  [0.2154751  0.18075559]
  [0.23373817 0.18768947]
  [0.24482158 0.18987371]]]

out2 is

[[[0.12221643 0.11449362]
  [0.18342148 0.1620608 ]
  [0.2154751  0.18075559]
  [0.23373817 0.18768945]
  [0.24482158 0.18987371]]]

EDIT:
I am running on the following configurations -

PyTorch - 1.0.1.post2
Python - 3.6.8 with GCC 7.3.0
OS - Pop!_OS 18.04 (Ubuntu 18.04, more-or-less)
CUDA - 9.1.85
Nvidia driver - 410.78

Output:
Initial weights for LSTM are small numbers close to 0, and by adding more layers the initial weighs and biases are getting smaller: all the weights and biases are initialized from -sqrt(k) to -sqrt(k), where k = 1/hidden_size (https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM)

By adding more layers you effectively multiply the input by many small numbers, so effect of the input is basically 0 and only biases in the later layers matter.

If you try LSTM with bias=False, you will see that output getting closer and closer to 0 with adding more layers.


Instruction: index selection in case of conflict in pytorch Argmax

Input:
I have been trying to learn tensor operations and this one has thrown me for a loop.
Let us say I have one tensor t:

    t = torch.tensor([
        [1,0,0,2],
        [0,3,3,0],
        [4,0,0,5]
    ], dtype  = torch.float32)

Now this is a rank 2 tensor and we can apply argmax for each rank/dimension.
let us say we apply it for dim = 1

t.max(dim = 1)
(tensor([2., 3., 5.]), tensor([3, 2, 3]))

Now we can see that the result is as expected the tensor along dim =1 has 2,3,and 5 as the max elements. But there is a conflict on 3. There are two values that are exactly similar.
How is it resolved? is it arbitrarily chosen? Is there an order for selecting like L-R, higher index value?
I'd appreciate any insights into how this is resolved!

Output:
That is a good question I stumbled over a couple of times myself. The simplest answer is that there are no guarantees whatsoever that torch.argmax (or torch.max(x, dim=k), which also returns indices when dim is specified) will return the same index consistently. Instead, it will return any valid index to the argmax value, possibly randomly. As this thread in the official forum discusses, this is considered to be desired behavior. (I know that there is another thread I read a while ago that makes this more explicit, but I cannot find it again).

Having said that, as this behavior was unacceptable to my usecase, I wrote the following functions that will find the left and rightmost indices (be aware that condition is a function-object you pass in):

def __consistent_args(input, condition, indices):
    assert len(input.shape) == 2, 'only works for batch x dim tensors along the dim axis'
    mask = condition(input).float() * indices.unsqueeze(0).expand_as(input)
    return torch.argmax(mask, dim=1)

def consistent_find_leftmost(input, condition):
    indices = torch.arange(input.size(1), 0, -1, dtype=torch.float, device=input.device)
    return __consistent_args(input, condition, indices)

def consistent_find_rightmost(input, condition):
    indices = torch.arange(0, input.size(1), 1, dtype=torch.float, device=input.device)
    return __consistent_args(input, condition, indices)

# one example:
consistent_find_leftmost(torch.arange(10).unsqueeze(0), lambda x: x>5)
# will return:
# tensor([6])

Hope they will help! (Oh, and please let me know if you have a better implementation that does the same)


Instruction: Convolutional encoder error - 'RuntimeError: input and target shapes do not match'

Input:
In below code three images are created, saved and a convolutional auto-encoder attempts to encode them to a lower dimensional representation.

%reset -f

import torch.utils.data as data_utils
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from matplotlib import pyplot as plt
from sklearn import metrics
import datetime
from sklearn.preprocessing import MultiLabelBinarizer
import seaborn as sns
sns.set_style("darkgrid")
from ast import literal_eval
import numpy as np
from sklearn.preprocessing import scale
import seaborn as sns
sns.set_style("darkgrid")
import torch
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable
from os import listdir
import cv2
import torch.nn.functional as F
import numpy as np
from numpy.polynomial.polynomial import polyfit
import matplotlib.pyplot as plt

number_channels = 3

%matplotlib inline

x = np.arange(10)
m = 1
b = 2
y = x * x
plt.plot(x, y)
plt.axis('off')
plt.savefig('1-increasing.jpg')

x = np.arange(10)
m = 0.01
b = 2
y = x * x * x
plt.plot(x, y)
plt.axis('off')
plt.savefig('2-increasing.jpg')

x = np.arange(10)
m = 0
b = 2
y = (m*x)+b
plt.plot(x, y)
plt.axis('off')
plt.savefig('constant.jpg')

batch_size_value = 2

train_image = []

train_image.append(cv2.imread('1-increasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('2-increasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('decreasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('constant.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))

data_loader = data_utils.DataLoader(train_image, batch_size=batch_size_value, shuffle=False,drop_last=True)

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.utils import save_image
from torchvision.datasets import MNIST
import os

def to_img(x):
    x = 0.5 * (x + 1)
    x = x.clamp(0, 1)
    x = x.view(x.size(0), 1, 28, 28)
    return x

num_epochs = 100
# batch_size = 128
batch_size = 2

learning_rate = 1e-3
dataloader = data_loader

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
#         torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True)
        self.encoder = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=3, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2),  # b, 16, 5, 5
            nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
            nn.ReLU(True),
            nn.MaxPool2d(3, stride=1)  # b, 8, 2, 2
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(8, 16, 2, stride=1),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 8, 3, stride=3, padding=1),  # b, 8, 15, 15
            nn.ReLU(True),
            nn.ConvTranspose2d(8, 3, 2, stride=2, padding=1),  # b, 1, 28, 28
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = autoencoder().cuda().double()

criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,
                             weight_decay=1e-5)

for epoch in range(num_epochs):
    for data in dataloader:
        img, _ = data
        img = img.double()
        img = Variable(img).cuda()
        img = img.unsqueeze_(0)

        # ===================forward=====================
        output = model(img)
        loss = criterion(output, img)
        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    # ===================log=================to_img=======
    print('epoch [{}/{}], loss:{:.4f}'
          .format(epoch+1, num_epochs, loss.data[0]))

torch.save(model.state_dict(), './conv_autoencoder.pth')

But error is returned :

RuntimeError: input and target shapes do not match: input [1 x 3 x 132 x 204], target [1 x 3 x 288 x 432] at /pytorch/aten/src/THCUNN/generic/MSECriterion.cu:15

The shape of the images are (3, 288, 432) . How to change the configuration of the model to allow [1 x 3 x 288 x 432] instead of [1 x 3 x 132 x 204] ?

Update:

I changed

nn.ConvTranspose2d(8, 3, 2, stride=2, padding=1)

to :

nn.ConvTranspose2d(8, 3, 3, stride=4, padding=2)

Which results in closer dimensional output but not exact so error is now :

RuntimeError: input and target shapes do not match: input [1 x 3 x 263 x 407], target [1 x 3 x 288 x 432] at /pytorch/aten/src/THCUNN/generic/MSECriterion.cu:12

How should the output decoder dimensions be calculated in order to produce the correct dimension ?

Output:
There are a couple of ways,
Here is the one solution:

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
#         torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True)
        self.encoder = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=3, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2),  # b, 16, 5, 5
            nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
            nn.ReLU(True),
            nn.MaxPool2d(3, stride=1)  # b, 8, 2, 2
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(8, 16, 2, stride=1),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 8, 3, stride=3, padding=1),  # b, 8, 15, 15
            nn.ReLU(True),
            nn.ConvTranspose2d(8, 3, 2, stride=2, padding=1),  # b, 1, 28, 28
            nn.ReLU(True),
            nn.ConvTranspose2d(3, 3, 2, stride=2, padding=1),  # b, 1, 28, 28
            nn.ReLU(True),
            nn.ConvTranspose2d(3, 3, 25, stride=1),
            nn.ReLU(True),
            nn.ConvTranspose2d(3, 3, 3, stride=1),
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

Here is the formula;

N --> Input Size, F --> Filter Size, stride-> Stride Size, pdg-> Padding size

ConvTranspose2d;

OutputSize = N*stride + F - stride - pdg*2

Conv2d;

OutputSize = (N - F)/stride + 1 + pdg*2/stride [e.g. 32/3=10 it ignores after the comma]


Instruction: How to do backprop in Pytorch (autograd.backward(loss) vs loss.backward()) and where to set requires_grad=True?

Input:
I have been using Pytorch for a while now. One question I had regarding backprop is as follows:

let's say we have a loss function for a neural network. For doing backprop, I have seen two different versions. One like:

optimizer.zero_grad()
autograd.backward(loss)
optimizer.step()

and the other one like:

optimizer.zero_grad()
loss.backward()
optimizer.step()

Which one should I use? Is there any difference between these two versions?

As a last question, do we need to specify the requires_grad=True for the parameters of every layer of our network to make sure their gradients is being computed in the backprop?

For example do I need to specify it for the layer nn.Linear(hidden_size, output_size) inside my network or it is automatically being set to True by default?

Output:
so just a quick answer: both autograd.backward(loss) and loss.backward() are actually the same. Just look at the implementation of tensor.backward() (as your loss is just a tensor), where tensor.loss just calls autograd.backward(loss).

As to your second question: whenever you use a prefabricated layer such as nn.Linear, or convolutions, or RNNs, etc., all of them rely on nn.Parameter attributes to store the parameters values. And, as the docs say, these default with requires_grad=True.

Update to a follow-up in the comments: To answer what happens to tensors when they are in a backward pass depends on whether a variable is on the computation path between the "output" and a leaf variable, or not. If not, it is not entirely clear what backprop should compute - after all, the entire purpose is to compute gradients for parameters, i.e., leaf-variables. If the tensor is on that path, all gradients will generally be automatically computed. For a more thorough discussion, see this question and this tutorial from the docs.


Instruction: Implementation of VGG16 on Pytorch giving size mismatch error

Input:
Snippet of my code implementation on PyTorch is:

model = models.vgg16(pretrained = False)
classifier = nn.Sequential(
        nn.Linear(25088, 128),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(128, 128),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(128, 20)
)
model.classifier = classifier

I'm feeding images of input size (60x60x3) and batch_size = 30.

When I run the code from Linux (Ubuntu) Terminal (with PyTorch Version:  1.0.0, Torchvision Version:  0.2.1) it gives me, the following error message:

  RuntimeError: size mismatch, m1: [30 x 512], m2: [25088 x 128]

While, when I run it from Spyder (Anaconda) on Windows (with PyTorch Version:  1.0.1, Torchvision Version:  0.2.2), it runs perfectly.

Am I missing something or is this because of some version mismatch in Pytorch and Torchvision? Both, I'm running on Python 3.6.
Please suggest.

[UPDATE: Mistakenly interchanged the version numbers for the error-case and error-free case. Thanks @Manoj Mohan for pointing it out]

Output:
It's probably the other way around. Things run perfectly on torchvision 0.2.2 and fails on torch vision 0.2.1.

This change of using AdaptiveAvgPool2d that went into 0.2.2 is why you don't see the error. https://github.com/pytorch/vision/commit/83b2dfb2ebcd1b0694d46e3006ca96183c303706

>>> import torch
>>> model = models.vgg16(pretrained = False)
>>> x = torch.randn(1,3,60,60) # random image
>>> feat = model.features(x)
>>> flat_feat = feat.view(feat.size(0), -1) # flatten
>>> flat_feat.shape
torch.Size([1, 512])
>>> model.classifier(flat_feat)

RuntimeError: size mismatch, m1: [1 x 512], m2: [25088 x 4096] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:940

You see the error of size mismatch. After, adaptive average pooling, things work fine.

>>> import torch.nn.functional as F
>>> avg = F.adaptive_avg_pool2d(feat, (7,7))
>>> avg = avg.view(avg.size(0), -1)
>>> output = model.classifier(avg)
>>> output.shape
torch.Size([1, 1000])


Instruction: Caffe2: Load ONNX model, and inference single threaded on multi-core host / docker

Input:
I'm having trouble running inference on a model in docker when the host has several cores. The model is exported via PyTorch 1.0 ONNX exporter:

torch.onnx.export(pytorch_net, dummyseq, ONNX_MODEL_PATH)

Starting the model server (wrapped in Flask) with a single core yields acceptable performance (cpuset pins the process to specific cpus) docker run --rm -p 8081:8080 --cpus 0.5 --cpuset-cpus 0 my_container

response from ab -c 1 -n 1000 http://0.0.0.0:8081/predict\?itemids\=5,100

Percentage of the requests served within a certain time (ms)
  50%      5
  66%      5
  75%      5
  80%      5
  90%      7
  95%     46
  98%     48
  99%     49

But pinning it to four cores gives completely different stats for the same ab-call docker run --rm -p 8081:8080 --cpus 0.5 --cpuset-cpus 0,1,2,3 my_container

Percentage of the requests served within a certain time (ms)
  50%      9
  66%     12
  75%     14
  80%     18
  90%     62
  95%     66
  98%     69
  99%     69
 100%     77 (longest request)

Model inference is done like this, and except this issue it seems to work as expected. (This runs in a completely separate environment from the model export of course)

from caffe2.python import workspace
from caffe2.python.onnx.backend import Caffe2Backend as c2
from onnx import ModelProto

class Model:
    def __init__(self):
        self.predictor = create_caffe2_predictor(path)

    @staticmethod
    def create_caffe2_predictor(onnx_file_path):
        with open(onnx_file_path, 'rb') as onnx_model:
            onnx_model_proto = ModelProto()
            onnx_model_proto.ParseFromString(onnx_model.read())
            init_net, predict_net = c2.onnx_graph_to_caffe2_net(onnx_model_proto)
            predictor = workspace.Predictor(init_net, predict_net)
        return predictor

    def predict(self, numpy_array):
        return self.predictor.run({'0': numpy_array})

** wrapper flask app which calls Model.predict() on calls to /predict **

OMP_NUM_THREADS=1 is also present in the container environment, which had some effect, but it is not the end issue.

The benchmark stats you're seeing here are run on a local machine with 8 hyperthreads, so I shouldn't be saturating my machine and affect the test. These results also show up in my kubernetes environment, and I'm getting a large amount of CFS (Completely Fair Scheduler) throttling there.

I'm running in a kubernetes environment, so there's no way for me to control how many CPUs the host exposes, and doing some sort of pinning there seems a bit hacky as well.

Is there any way to pin caffe2 model inference to a single processor? Am I doing something obviously wrong here? Is the caffe2.Predictor object not suited to this task?

Any help appreciated.

EDIT:

I've added the simplest possible reproducable example I can think of here, with a docker-container and run-script included: https://github.com/NegatioN/Caffe2Struggles

Output:
This is not a direct answer to the question, but if your goal is to serve PyTorch models (and only PyTorch models, as mine is now) in production, simply using PyTorch Tracing seems to be the better choice.

You can then load it directly into a C++ frontend similarly to what you would do through Caffe2, but PyTorch tracing seems more well maintained. From what I can see there are no speed slowdowns, but it is a whole lot easier to configure.

An example of this to get good performance on a single-core container is to run with OMP_NUM_THREADS=1 as before, and export the model as follows:

from torch import jit
### Create a model
model.eval()
traced = jit.trace(model, torch.from_numpy(an_array_with_input_size))
traced.save("traced.pt")

And then simply run the model in production in pure C++ following the above guide, or through the Python interface as such:

from torch import jit
model = jit.load("traced.pt")
output = model(some_input)


Instruction: Convert a torch t7 model to keras h5

Input:
How can we convert a t7 model to keras' h5 ?
I am trying to do so for c3d-sports1m-kinetics.t7 that you can find here https://github.com/kenshohara/3D-ResNets/releases

The least I can ask for is a way to load the t7 model to python (pytorch) and then extract its weights, but I couldn't do it with the load_lua() function...

I get an error while trying to do it using this function https://github.com/pytorch/pytorch/blob/c6529f4851bb8ac95f05d3f17dea178a0367aaee/torch/utils/serialization/read_lua_file.py

The error that I get is the following :

Traceback (most recent call last):
File "convert_t7_to_hdf5.py", line 574, in <module>
    a = load_lua("model.t7")
  File "convert_t7_to_hdf5.py", line 571, in load_lua
    return reader.read()
  File "convert_t7_to_hdf5.py", line 542, in read
    typeidx = self.read_int()
  File "convert_t7_to_hdf5.py", line 440, in read_int
    return self._read('i')
  File "convert_t7_to_hdf5.py", line 431, in _read
    result = struct.unpack(fmt, self.f.read(sz))
ValueError: read of closed file

Output:
As mentioned in this link,
https://github.com/pytorch/pytorch/issues/15307#issuecomment-448086741

with torchfile package, the load was successful. You can dump the contents of model to a file and then understand the contents. Each layer information is stored as a dictionary. Knowing the model architecture would make it easier to parse the contents.

>>> import torchfile
>>> model = torchfile.load('c3d-sports1m-kinetics.t7')
>>> module = model.modules[0].modules[0]
>>> module.name
b'conv1a'
>>> module['weight'].shape
(64, 3, 3, 3, 3)
>>> module['bias'].shape
(64,)


Instruction: Invalid combination of arguments - eq()

Input:
I'm using a code shared here to test a CNN image classifier. When I call the test function, I got this error on line 155:

test_acc += torch.sum(prediction == labels.data)
TypeError: eq() received an invalid combination of arguments - got (numpy.ndarray), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: ([31;1mnumpy.ndarray[0m)
 * (Number other)
      didn't match because some of the arguments have invalid types: ([31;1mnumpy.ndarray[0m)

Fragment of the test function:

def test():
    model.eval()
    test_acc = 0.0
    for i, (images, labels) in enumerate(test_loader):

        if cuda_avail:
                images = Variable(images.cuda())
                labels = Variable(labels.cuda())

        #Predict classes using images from the test set
        outputs = model(images)
        _,prediction = torch.max(outputs.data, 1)
        prediction = prediction.cpu().numpy()
        test_acc += torch.sum(prediction == labels.data) #line 155

    #Compute the average acc and loss over all 10000 test images
    test_acc = test_acc / 10000

return test_acc

After a quick search I see that the error is probably related to the comparison between the prediction and labels, as seem in this SO question.

Any idea on how to fix this?

Output:
Why do you have .numpy() here prediction = prediction.cpu().numpy()?
That way you convert PyTorch tensor to NumPy array, making it incompatible type to compare with labels.data.

Removing .numpy() part should fix the issue.


Instruction: How to pass parameters to forward function of my torch nn.module from skorch.NeuralNetClassifier.fit()

Input:
I have extended nn.Module to implement my network whose forward function is like this ...

def forward(self, X, **kwargs):

    batch_size, seq_len = X.size()

    length = kwargs['length']
    embedded = self.embedding(X) # [batch_size, seq_len, embedding_dim]
    if self.use_padding:
        if length is None:
            raise AttributeError("Length must be a tensor when using padding")
        embedded = nn.utils.rnn.pack_padded_sequence(embedded, length, batch_first=True)
        #print("Size of Embedded packed", embedded[0].size())

    hidden, cell = self.init_hidden(batch_size)
    if self.rnn_unit == 'rnn':
        out, _ = self.rnn(embedded, hidden)
    elif self.rnn_unit == 'lstm':
        out, (hidden, cell) = self.rnn(embedded, (hidden, cell))

    # unpack if padding was used
    if self.use_padding:
        out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first = True)

I initialized a skorch NeuralNetClassifier like this,

net = NeuralNetClassifier(
    model,
    criterion=nn.CrossEntropyLoss,
    optimizer=Adam,
    max_epochs=8,
    lr=0.01,
    batch_size=32
)

Now if I call net.fit(X, y, length=X_len) it throws an error

TypeError: __call__() got an unexpected keyword argument 'length'

According to the documentation fit function expects a fit_params dictionary,

**fit_params : dict
   Additional parameters passed to the ``forward`` method of
   the module and to the ``self.train_split`` call.

and the source code always send my parameters to train_split where obviously my keyword argument would not be recognized.

Is there any way around to pass the arguments to my forward function?

Output:
The fit_params parameter is intended for passing information that is relevant to data splits and the model alike, like split groups.

In your case, you are passing additional data to the module via fit_params which is not what it is intended for. In fact, you could easily run into trouble doing this if you, for example, enable batch shuffling on the train data loader since then your lengths and your data are misaligned.

The best way to do this is already described in the answer to your question on the issue tracker:

X_dict = {'X': X, 'length': X_len}
net.fit(X_dict, y)

Since skorch supports dicts you can simply add the length's to your input dict and have it both passed to the module, nicely batched and passed through the same data loader. In your module you can then access it via the parameters in forward:

def forward(self, X, length):
     return ...

Further documentation of this behaviour can be found in the docs.


Instruction: U-net with pre-trained backbones: where to make skip connections?

Input:
I'm trying to implement U-Net with PyTorch, using pre-trained networks in the encoder path.

The original U-Net paper trained a network from scratch. Are there any resources or principles on where skip connections should be placed, when a pre-trained backbone is used instead?

I already found some examples (e.g. this repo), but without any justification for the feature selection.

Output:
In the original U-Net paper features right before Max-Pool layers are used for the skip-connections.

The logic is exactly the same with pre-trained backbones: at each spatial resolution, the deepest feature layer is selected. Thanks for qubvel on github for pointing this out in an issue.


Instruction: Display examples of augmented images in PyTorch

Input:
I want to display some samples of augmented training images.

My transform includes the standard ImageNet transforms.Normalize like this:

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406],
                                                            [0.229, 0.224, 0.225])])

However, because of the Normalise, the images display in weird colours.

This answer says I'd need access to the original image, which is difficult when the transforms are applied at load time:

image_datasets['train'] = datasets.ImageFolder(train_dir, transform=train_transforms)

How would I go about displaying a few sample augmented images in their usual colours while using the normalised ones for calculation?

Output:
To answer my own question, I came up with the following:

# Undo transforms.Normalize
def denormalise(image):
    image = image.numpy().transpose(1, 2, 0)  # PIL images have channel last
    mean = [0.485, 0.456, 0.406]
    stdd = [0.229, 0.224, 0.225]
    image = (image * stdd + mean).clip(0, 1)
    return image

example_rows = 2
example_cols = 5

sampler = torch.utils.data.RandomSampler(image_datasets['train'],
                                         num_samples=example_rows * example_cols)

# Get a batch of images and labels
images, indices = next(iter(sampler))

plt.rcParams['figure.dpi'] = 120  # Increase size of pyplot plots

# Show a grid of example images
fig, axes = plt.subplots(example_rows, example_cols, figsize=(9, 5)) #  sharex=True, sharey=True)
axes = axes.flatten()
for ax, image, index in zip(axes, images, indices):
    ax.imshow(denormalise(image))
    ax.set_axis_off()
    ax.set_title(class_names[index], fontsize=7)

fig.subplots_adjust(wspace=0.02, hspace=0)
fig.suptitle('Augmented training set images', fontsize=20)
plt.show()

This is based on PyTorch's Transfer Learning Tutorial's code but displays the title above each image and generally looks much nicer.


Instruction: LSTMCell parameters is not shown Pytorch

Input:
I have the following code:

class myLSTM(nn.Module):
def __init__(self, input_size, output_size, hidden_size, num_layers):
    super(myLSTM, self).__init__()
    self.input_size = input_size + 1
    self.output_size = output_size
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.layers = []
    new_input_size = self.input_size
    for i in xrange(num_layers):
        self.layers.append(LSTMCell(new_input_size, hidden_size))
        new_input_size = hidden_size
    self.linear = nn.Linear(hidden_size, output_size)
    self.softmax = nn.Softmax()

def forwardLayers(self, input, hns, cns, layers):
    new_hns = []
    new_cns = []
    (hn, cn) = layers[0](input, (hns[0], cns[0]))
    new_hns.append(hn)
    new_cns.append(cn)
    for i in xrange(1, len(layers)):
        (hn, cn) = layers[i](hn, (hns[i], cns[i]))
        new_hns.append(hn)
        new_cns.append(cn)
    return hn, (new_hns, new_cns)

def forward(self, input, hx):
    actions = []
    hns, cns = hx
    action = torch.Tensor([[0.0]])
    for i in range(len(input)):
        new_input = input[i]
        new_input = new_input.view(1, -1)
        output, (hns, cns) = self.forwardLayers(new_input, hns, cns, self.layers)
        output = self.softmax(self.linear(output))

    return output

Now when I call the following code to see the parameters of my network:

for name, param in myLSTM_object.named_parameters():
        if param.requires_grad:
            print name, param.data

What I get is:

linear.weight tensor([[ 0.5042, -0.6984],
    [ 0.0721, -0.4060]])
linear.bias tensor([ 0.6968, -0.4649])

So, it completely misses the parameters of LSTMCell. Does this mean the parameters of LSTMCell is not trained. What should I do to see LSTMCell parameters?

Output:
This is to be expected - storing modules in list, dict, set or other python containers does not register them with the module owning said list, etc. To make your code work, use nn.ModuleList instead. It's as simple as modifying your __init__ code to use

layers = []
new_input_size = self.input_size
for i in xrange(num_layers):
    layers.append(LSTMCell(new_input_size, hidden_size))
    new_input_size = hidden_size
self.layers = nn.ModuleList(layers)


Instruction: Understanding PyTorch training batches

Input:
Reading https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel & https://discuss.pytorch.org/t/how-does-enumerate-trainloader-0-work/14410 I'm trying to understand how training epochs behave in PyTorch.

Take this outer and inner loop :

for epoch in range(num_epochs):
    for i1,i2 in enumerate(training_loader):

Is this a correct interpretation : ?

For each invocation of the outer loop/epoch the entire training set, in above example training_loader is iterated per batch. This means the model does not process one instance per training cycle. Per training cycle ( for epoch in range(num_epochs): ) the entire training set is processed in chunks/batches where the batch size is determined when creating training_loader

Output:
torch.utils.data.DataLoader returns an iterable that iterates over the dataset.

Therefore, the following -

training_loader = torch.utils.data.DataLoader(*args)
for i1,i2 in enumerate(training_loader):

  #process

runs one over the dataset completely in batches.


Instruction: Understanding when to use python list in Pytorch

Input:
Basically as this thread discusses here, you cannot use python list to wrap your sub-modules (for example your layers); otherwise, Pytorch is not going to update the parameters of the sub-modules inside the list. Instead you should use nn.ModuleList to wrap your sub-modules to make sure their parameters are going to be updated. Now I have also seen codes like following where the author uses python list to calculate the loss and then do loss.backward() to do the update (in reinforce algorithm of RL). Here is the code:

 policy_loss = []
    for log_prob in self.controller.log_probability_slected_action_list:
        policy_loss.append(- log_prob * (average_reward - b))
    self.optimizer.zero_grad()
    final_policy_loss = (torch.cat(policy_loss).sum()) * gamma
    final_policy_loss.backward()
    self.optimizer.step()

Why using the list in this format works for updating the parameters of modules but the first case does not work? I am very confused now. If I change in the previous code policy_loss = nn.ModuleList([]), it throws an exception saying that tensor float is not sub-module.

Output:
You are misunderstanding what Modules are. A Module stores parameters and defines an implementation of the forward pass.

You're allowed to perform arbitrary computation with tensors and parameters resulting in other new tensors. Modules need not be aware of those tensors. You're also allowed to store lists of tensors in Python lists. When calling backward it needs to be on a scalar tensor thus the sum of the concatenation. These tensors are losses and not parameters so they should not be attributes of a Module nor wrapped in a ModuleList.


Instruction: Pytorch select values from the last tensor dimension with indices from another tenor with a smaller dimension

Input:
I have a tensor a with three dimensions. The first dimension corresponds to minibatch size, the second to the sequence length, and the third to the feature dimension. E.g.,

>>> a = torch.arange(1, 13, dtype=torch.float).view(2,2,3)  # Consider the values of a to be random
>>> a
tensor([[[ 1.,  2.,  3.],
         [ 4.,  5.,  6.]],

        [[ 7.,  8.,  9.],
         [10., 11., 12.]]])

I have a second, two-dimensional tensor. Its first dimension corresponds to the minibatch size and its second dimension to the sequence length. It contains values in the range of the indices of the third dimension of a. as third dimension has size 3, so b can contain values 0, 1 or 2. E.g.,

>>> b = torch.LongTensor([[0, 2],[1,0]])
>>> b
tensor([[0, 2],
        [1, 0]])

I want to obtain a tensor c that has the shape of b and contains all the values of a that are referenced by b.
In the upper scenario I would like to have:

c = torch.empty(2,2)
c[0,0] = a[0, 0, b[0,0]]
c[1,0] = a[1, 0, b[1,0]]
c[0,1] = a[0, 1, b[0,1]]
c[1,1] = a[1, 1, b[1,1]]

>>> c
tensor([[ 1.,  5.],
        [ 8., 10.]])

How can I create the tensor c fast? Further, I also want c to be differentiable (be able to use .backprob()). I am not too familiar with pytorch, so I am not sure, if a differentiable version of this exists.

As an alternative, instead of c having the same shape as b I could also use a c with the same shape of a, having only zeros, but at the places referenced by b ones. Then I could multiply a and c to obtain a differentiable tensor.

Like follows:

c = torch.zeros(2,2,3, dtype=torch.float)
c[0,0,b[0,0]] = 1
c[1,0,b[1,0]] = 1
c[0,1,b[0,1]] = 1
c[1,1,b[1,1]] = 1

>>> a*c
tensor([[[ 1.,  0.,  0.],
         [ 0.,  5.,  0.]],

        [[ 0.,  8.,  0.],
         [10.,  0.,  0.]]])

Output:
Lets declare necessary variables first: (notice requires_grad in a's initialization, we will use it to ensure differentiability)

a = torch.arange(1,13,dtype=torch.float32,requires_grad=True).reshape(2,2,3)
b = torch.LongTensor([[0, 2],[1,0]])

Lets reshape a and squash minibatch and sequence dimensions:

temp = a.reshape(-1,3)

so temp now looks like:

tensor([[ 1.,  2.,  3.],
    [ 4.,  5.,  6.],
    [ 7.,  8.,  9.],
    [10., 11., 12.]], grad_fn=<AsStridedBackward>)

Notice now each value of b can be used in each row of temp to get desired output. Now we do:

c = temp[range(len(temp )),b.view(-1)].view(b.size())

Notice how we index temp, range(len(temp )) to select each row and 1D b i.e b.view(-1) to get corresponding columns. Lastly .view(b.size()) brings this array to the same size as b.

If we print c now:

tensor([[ 1.,  6.],
    [ 8., 10.]], grad_fn=<ViewBackward>)

The presence of grad_fn=.. shows that c requires gradient i.e. its differentiable.


Instruction: Inconsistency when comparing scipy, torch and fourier periodic convolution

Input:
I'm implementing a 2d periodic convolution on a synthetic image in three different ways: using scipy, using torch and using the Fourier transform (also under torch framework).

However, I've got different results. Performing the operation by hand I can see that scipy's convolution yields the correct results. torch's spatial version, on the other hand, yields the expected result inverted. Finally, the Fourier version returns something unexpected.

The code is the following:

import torch
import numpy as np
import scipy.signal as sig
import torch.nn.functional as F
import matplotlib.pyplot as plt

def numpy_periodic_conv(f, k):
    H, W = f.shape
    periodic_f = np.hstack([f, f])
    periodic_f = np.vstack([periodic_f, periodic_f])
    conv = sig.convolve2d(periodic_f, k, mode='same')
    conv = conv[H // 2:-H // 2, W // 2:-W // 2]
    return periodic_f, conv

def torch_periodic_conv(f, k):
    H, W = f.shape[-2:]
    periodic_f = f.repeat(1, 1, 2, 2)
    conv = F.conv2d(periodic_f, k, padding=1)
    conv = conv[:, :, H // 2:-H // 2, W // 2:-W // 2]
    return periodic_f.squeeze().numpy(), conv.squeeze().numpy()

def torch_fourier_conv(f, k):
    pad_x = f.shape[-2] - k.shape[-2]
    pad_y = f.shape[-1] - k.shape[-1]
    expanded_kernel = F.pad(k, [0, pad_x, 0, pad_y])
    fft_x = torch.rfft(f, 2, onesided=False)
    fft_kernel = torch.rfft(expanded_kernel, 2, onesided=False)
    real = fft_x[:, :, :, :, 0] * fft_kernel[:, :, :, :, 0] - \
           fft_x[:, :, :, :, 1] * fft_kernel[:, :, :, :, 1]
    im = fft_x[:, :, :, :, 0] * fft_kernel[:, :, :, :, 1] + \
         fft_x[:, :, :, :, 1] * fft_kernel[:, :, :, :, 0]
    fft_conv = torch.stack([real, im], -1) # (a+bj)*(c+dj) = (ac-bd)+(ad+bc)j
    ifft_conv = torch.irfft(fft_conv, 2, onesided=False)
    return expanded_kernel.squeeze().numpy(), ifft_conv.squeeze().numpy()

if __name__ == '__main__':
    f = np.concatenate([np.ones((10, 5)), np.zeros((10, 5))], 1)
    k = np.array([[1, 0, -1], [2, 0, -2], [1, 0, -1]])

    f_tensor = torch.from_numpy(f).unsqueeze(0).unsqueeze(0).float()
    k_tensor = torch.from_numpy(k).unsqueeze(0).unsqueeze(0).float()

    np_periodic_f, np_periodic_conv = numpy_periodic_conv(f, k)
    tc_periodic_f, tc_periodic_conv = torch_periodic_conv(f_tensor, k_tensor)
    tc_fourier_k, tc_fourier_conv = torch_fourier_conv(f_tensor, k_tensor)

    print('Spatial numpy conv shape= ', np_periodic_conv.shape)
    print('Spatial torch conv shape= ', tc_periodic_conv.shape)
    print('Fourier torch conv shape= ', tc_fourier_conv.shape)

    r_np = dict(name='numpy', im=np_periodic_f, k=k, conv=np_periodic_conv)
    r_torch = dict(name='torch', im=tc_periodic_f, k=k, conv=tc_periodic_conv)
    r_fourier = dict(name='fourier', im=f, k=tc_fourier_k, conv=tc_fourier_conv)
    titles = ['{} im', '{} kernel', '{} conv']
    results = [r_np, r_torch, r_fourier]
    fig, axs = plt.subplots(3, 3)
    for i, r_dict in enumerate(results):
        axs[i, 0].imshow(r_dict['im'], cmap='gray')
        axs[i, 0].set_title(titles[0].format(r_dict['name']))
        axs[i, 1].imshow(r_dict['k'], cmap='gray')
        axs[i, 1].set_title(titles[1].format(r_dict['name']))
        axs[i, 2].imshow(r_dict['conv'], cmap='gray')
        axs[i, 2].set_title(titles[2].format(r_dict['name']))
    plt.show()

The results I'm obtaining:

Note: The image for both numpyand torch versions show the periodic image, which is required to perform the periodic convolution. The kernel for the Fourier version shows the original kernel zero-padded to the image size, which is required to compute the element-wise multiplication in the frequency domain.

-Edit1: There was an error when in the multiplication in the Fourier version, I was doing (ac-bd)+(ad-bc)j instead of (ac-bd)+(ad+bc)j. But now, I get the convolution shifted by one column.

-Edit2: torch's spatial convolution results is inverted because the operation is actually a cross-correlation. This was confirmed in the pytorch's official forum here. Furthermore, after fixing the kernel padding as Cris Luengo's answer, the frequency method yielded the same results as the correlations. This is rather strange for me because, as far as I know, the frequency property hold for convolution, not correlation.

New-results after fixing the kernel:

Output:
The FFT result is wrong because the padding is wrong. When padding, you need to put the origin (center of the kernel) at the top-left corner of the image. See this other answer for details.
The difference between the other two is the difference between a convolution and a correlation. It looks like the “numpy“ result is a convolution, the “torch” one a correlation.


Instruction: Unable to Normalize Tensor in PyTorch

Input:
I am trying to normalize the tensor outputted by my network but am getting an error in doing so. The code is as follows:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model_load_path = r'path\to\saved\model\file'
model.load_state_dict(torch.load(model_load_path))
model.eval()
output = model(input).to(device).view(-1, 1, 150, 150)
inv_normalize = transforms.Compose(
    [
        transforms.Normalize(mean=[-0.5/0.5], std=[1/0.5])
    ]
)
print(output.size())  # The size printed is torch.Size([1, 1, 150, 150])
output = inv_normalize(output)

I am getting an error in the following line:
output = inv_normalize(output)

The error reads:
TypeError: tensor is not a torch image.

My output is a single image, with a single channel, and height and width = 150
Any help will be appreciated! Thanks!

Output:
In order to apply transforms.Normalize you have to convert the input to a tensor. For this you can use transforms.ToTensor.
inv_normalize = transforms.Compose(
    [
        transforms.toTensor(),
        transforms.Normalize(mean=[-0.5/0.5], std=[1/0.5])
    ]
)

This tensor must consist of three dimensions (channels, height, width). Currently you have one dimension too much. Just remove the extra dimension in your view call:
output = model(input).to(device).view(1, 150, 150)


Instruction: Load a plain text file into PyTorch

Input: I have two separate files, one is a text file, with each line being a single text. The other file contains the class label of that corresponding line. How do I load this into PyTorch and carry out further tokenization, embedding, etc?

Output:
What have you tried already? What you described is still not very PyTorch related, you can make a pre-processing script that loads all the sentences into single data structured, e.g.: a list of (text, label) tuple.You can also already split your data into training and hold-out set in this step. You can then dump all this into .csv files.

Then, one way to do it is in 3 steps:

Implement the class Dataset - to load efficiently your data, reading the produced .csv files;
Have another like Vocabulary that keeps a mapping from tokens to ids and vice-verse;
Something like a Vectorizer, that converts your sentences into vectors, either one-hot-encondings or embeddings;

Then you can use this to produce a vector representation of your sentences a pass it to a neural network.

Look into this notebook to understand all this in more detail:

Sentiment Classification


Instruction: Common class for Linear, Conv1d, Conv2d,..., LSTM,

Input:
Is there any class that all torch::nn::Linear, torch::nn::Conv1d, torch::nn::Conv2d, ... torch::nn::GRU, .... all inherit from that? torch::nn::Module seems be a good option, though there is a middle class, called torch::nn::Cloneable, so that torch::nn::Module does not work. Also, torch::nn::Cloneable itself is a template so that needs type in the declaration.
I want to create a general class model, which has std::vector<the common class> layers, so that later I can fill layers with any type of layer that I want, e.g., Linear, LSTM, etc. Is there such a capability in the current API? This can be done easily in python, though here we need declaration and this hinders the python's easiness.

Thanks,
Afshin

Output:
I found that nn::sequential can be used for a this purpose, and it does not need a forward implementation, which can be a positive point and at a same time a negative point. nn::sequential already requires each module to have a forward implementation, and calls the forward functions in a sequence that they have added in. So, one cannot create an ad-hock non-usual forward pass like Dense-Net with that, though it is good enough for general usages.

In addition, it seems that nn::sequential just uses a std::vector<nn::AnyModule> as its underlying module list. So, std::vector<nn::AnyModule> also might be used.


Instruction: Encountering Import Error DLL load failed constantly

Input:
I have been trying to intall scikit-learn and pytorch using their respective commands given in the docs:
The commands for installing PyTorch are:
1) pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.1-cp37-cp37m-win_amd64.whl
2) pip3 install torchvision

The command for installing scikit-learn is:
pip install -U scikit-learn

Some background:
I am using Windows 8.1, Python 3.7.2. My pip is updated. I have also installed Anaconda for solving this using conda, but had zero luck!(Also, here I am running into 'conda' unrecognized error which is another story). Here are the paths my PATH variable holds.
PATH

C:\Users\satya\Anaconda3;
C:\Users\satya\Anaconda3\Library\mingw-w64\bin;
C:\Users\satya\Anaconda3\Library\usr\bin;
C:\Users\satya\Anaconda3\Library\bin;
C:\Users\satya\Anaconda3\Scripts;
C:\Users\satya\AppData\Local\Programs\Python\Python37\Scripts\;
C:\Users\satya\AppData\Local\Programs\Python\Python37\; C:\Users\satya\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Python 3.7

The Actual Problem:
The same commands for installation given above work perfectly fine on my other Windows 10, but, for my Windows 8.1 it gives this error which has become a real PITA

Import Error: DLL load failed The specified module could not be found

When I import sklearn or import torch I get the exact same error. All the time.
Back Story:
I have searched almost all the related questions I could find on Stackoverflow and Github for 6+ hours to help me solve this problem. But, none of the answers have helped till now and some haven't had an "understandable" answer. Maybe, its just a small fix, but now, I am choosing to post a question on SO.
My Question Again:
Can someone please help out and try to explain what I am missing out here? I really want to fix this error for good(and want to be in a position to fix it if I encounter it again). An elaborate answer would really help understand easily.
Thank You!

Output:
Please check your python build number with the following command.

conda list python

Python 3.7.2 with build number h8c8aaf0_2 has a solved issue.

If this is the case, an update will do.

conda update python