Instruction: Neural network versus random forest performance discrepancy

Input:
I want to run some experiments with neural networks using PyTorch, so I tried a simple one as a warm-up exercise, and I cannot quite make sense of the results.

The exercise attempts to predict the rating of 1000 TPTP problems from various statistics about the problems such as number of variables, maximum clause length etc. Data file https://github.com/russellw/ml/blob/master/test.csv is quite straightforward, 1000 rows, the final column is the rating, started off with some tens of input columns, with all the numbers scaled to the range 0-1, I progressively deleted features to see if the result still held, and it does, all the way down to one input column; the others are in previous versions in Git history.

I started off using separate training and test sets, but have set aside the test set for the moment, because the question about whether training performance generalizes to testing, doesn't arise until training performance has been obtained in the first place.

Simple linear regression on this data set has a mean squared error of about 0.14.

I implemented a simple feedforward neural network, code in https://github.com/russellw/ml/blob/master/test_nn.py and copied below, that after a couple hundred training epochs, also has an mean squared error of 0.14.

So I tried changing the number of hidden layers from 1 to 2 to 3, using a few different optimizers, tweaking the learning rate, switching the activation functions from relu to tanh to a mixture of both, increasing the number of epochs to 5000, increasing the number of hidden units to 1000. At this point, it should easily have had the ability to just memorize the entire data set. (At this point I'm not concerned about overfitting. I'm just trying to get the mean squared error on training data to be something other than 0.14.) Nothing made any difference. Still 0.14. I would say it must be stuck in a local optimum, but that's not supposed to happen when you've got a couple million weights; it's supposed to be practically impossible to be in a local optimum for all parameters simultaneously. And I do get slightly different sequences of numbers on each run. But it always converges to 0.14.

Now the obvious conclusion would be that 0.14 is as good as it gets for this problem, except that it stays the same even when the network has enough memory to just memorize all the data. But the clincher is that I also tried a random forest, https://github.com/russellw/ml/blob/master/test_rf.py

... and the random forest has a mean squared error of 0.01 on the original data set, degrading gracefully as features are deleted, still 0.05 on the data with just one feature.

Nowhere in the lore of machine learning is it said 'random forests vastly outperform neural nets', so I'm presumably doing something wrong, but I can't see what it is. Maybe it's something as simple as just missing a flag or something you need to set in PyTorch. I would appreciate it if someone could take a look.

import numpy as np
import pandas as pd
import torch
import torch.nn as nn

# data
df = pd.read_csv("test.csv")
print(df)
print()

# separate the output column
y_name = df.columns[-1]
y_df = df[y_name]
X_df = df.drop(y_name, axis=1)

# numpy arrays
X_ar = np.array(X_df, dtype=np.float32)
y_ar = np.array(y_df, dtype=np.float32)

# torch tensors
X_tensor = torch.from_numpy(X_ar)
y_tensor = torch.from_numpy(y_ar)

# hyperparameters
in_features = X_ar.shape[1]
hidden_size = 100
out_features = 1
epochs = 500

# model
class Net(nn.Module):
    def __init__(self, hidden_size):
        super(Net, self).__init__()
        self.L0 = nn.Linear(in_features, hidden_size)
        self.N0 = nn.ReLU()
        self.L1 = nn.Linear(hidden_size, hidden_size)
        self.N1 = nn.Tanh()
        self.L2 = nn.Linear(hidden_size, hidden_size)
        self.N2 = nn.ReLU()
        self.L3 = nn.Linear(hidden_size, 1)

    def forward(self, x):
        x = self.L0(x)
        x = self.N0(x)
        x = self.L1(x)
        x = self.N1(x)
        x = self.L2(x)
        x = self.N2(x)
        x = self.L3(x)
        return x

model = Net(hidden_size)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

# train
print("training")
for epoch in range(1, epochs + 1):
    # forward
    output = model(X_tensor)
    cost = criterion(output, y_tensor)

    # backward
    optimizer.zero_grad()
    cost.backward()
    optimizer.step()

    # print progress
    if epoch % (epochs // 10) == 0:
        print(f"{epoch:6d} {cost.item():10f}")
print()

output = model(X_tensor)
cost = criterion(output, y_tensor)
print("mean squared error:", cost.item())

Output:
can you please print the shape of your input ?
I would say check those things first:

that your target y have the shape (-1, 1) I don't know if pytorch throws an Error in this case. you can use y.reshape(-1, 1) if it isn't 2 dim
your learning rate is high. usually when using Adam the default value is good enough or try simply to lower your learning rate. 0.1 is a high value for a learning rate to start with
place the optimizer.zero_grad at the first line inside the for loop
normalize/standardize your data ( this is usually good for NNs )
remove outliers in your data (my opinion: I think this can't affect Random forest so much but it can affect NNs badly)
use cross validation (maybe skorch can help you here. It's a scikit learn wrapper for pytorch and easy to use if you know keras)

Notice that Random forest regressor or any other regressor can outperform neural nets in some cases. There is some fields where neural nets are the heros like Image Classification or NLP but you need to be aware that a simple regression algorithm can outperform them. Usually when your data is not big enough.


Instruction: Masking and computing loss for a padded batch sent through an RNN with a linear output layer in pytorch

Input:
Although a typical use case, I can't find one simple and clear guide on what is the canonical way to compute loss on a padded minibatch in pytorch, when sent through an RNN.

I think a canonical pipeline could be:

1) The pytorch RNN expects a padded batch tensor of shape: (max_seq_len, batch_size, emb_size)

2) So we give an Embedding layer for example this tensor:

tensor([[1, 1],
        [2, 2],
        [3, 9]])

9 is the padding index. Batch size is 2. The Embedding layer will make it to be of shape (max_seq_len, batch_size, emb_size). The sequences in the batch are in descending order, so we can pack it.

3) We apply pack_padded_sequence, we apply the RNN, finally we apply pad_packed_sequence. We have at this point (max_seq_len, batch_size, hidden_size)

4) Now we apply the linear output layer on the result and let's say the log_softmax. So at the end we have a tensor for a batch of scores of shape: (max_seq_len, batch_size, linear_out_size)

How should I compute the loss from here, masking out the padded part (with an arbitrary target)? Thanks!

Output:
I think the PyTocrh Chatbot Tutorial might be instructional for you.

Basically, you calculate the mask of valid output values (paddings are not valid), and use that to calculate the loss for only those values.

See the outputVar and maskNLLLoss methods on the tutorial page. For your convenience I copied the code here, but you really need to see it in context of all the code.

# Returns padded target sequence tensor, padding mask, and max target length
def outputVar(l, voc):
    indexes_batch = [indexesFromSentence(voc, sentence) for sentence in l]
    max_target_len = max([len(indexes) for indexes in indexes_batch])
    padList = zeroPadding(indexes_batch)
    mask = binaryMatrix(padList)
    mask = torch.BoolTensor(mask)
    padVar = torch.LongTensor(padList)
    return padVar, mask, max_target_len

def maskNLLLoss(inp, target, mask):
    nTotal = mask.sum()
    crossEntropy = -torch.log(torch.gather(inp, 1, target.view(-1, 1)).squeeze(1))
    loss = crossEntropy.masked_select(mask).mean()
    loss = loss.to(device)
    return loss, nTotal.item()


Instruction: Can i have mutliple labels for a single image in an image classification scenario

Input:
If I am building a model where I need to predict the vehicle, color of it, and make of it, then can I use all the labels for a single image and build my model around it.

Like for a single image of a vehicle which is a car (car1.jpg) will have labels like - Sedan(Make), Blue(Color) and Car(Type of vehicle). Can I make a single model for this or I will have to make 3 separate models for this problem.

Output:
You can have a single model with multiple outputs. As you seem interested with image processing, detection models for example (like SSD, RFCN, etc.) have multiple outputs, one for classes, one for box coordinates. Take a look a the page 3 of this article for the "feature extractor"/"classifier" split.

In fact, you will have a first common part of your model (with mostly the convolution layers to extract your features).
Deeper in your model, you will have separate parts, one for each kind of prediction.


Instruction: different between class name vs self calling in super()

Input:
isn't it same meaning of class name NeuralNet and self keyword passing in super call  -
super(NeuralNet,self).__init__() # init super

here is the code snippet from example:

class NeuralNet(nn.Module):
    def __init__(self, use_batch_norm, input_size=784, hidden_dim=256, output_size=10):
        """
        Creates a PyTorch net using the given parameters.
        """
        super(NeuralNet, self).__init__() # init super
        # continues code

Output:
Given your question, I kindly but very strongly suggest you do the full official Python tutorial.

And no, NeuralNet and self are NOT the same thing. The first is the NeuralNet class, the second is the current NeuralNet instance ("current": the one on which the method has been called).


Instruction: Attribute error: 'ReduceLROnPlateau' object has no attribute _cmp

Input:
I'm trying to load an existing pytorch model with -

from mymodels import model
m = torch.load("model.pth", map_location=torch.device("cpu"))

I'm getting the error -

  line 613, in _load

  result = unpickler.load()

  Attribute error: 'ReduceLROnPlateau' object has no attribute _cmp

my pytorch version is 1.3.1. How do I fix this?

Output: This was because of a version difference between the pytorch the network is trained on and the pytorch being used. It was resolved by using the same versions.


Instruction: Pytorch: Batch size is missing in data after torch.utils.random_split() is used on dataloader.dataset

Input:
I used random_split() to divide my data into train and test and I observed that if random split is done after the dataloader is created, batch size is missing when getting a batch of data from the dataloader.

import torch
from torchvision import transforms, datasets
from torch.utils.data import random_split

# Normalize the data
transform_image = transforms.Compose([
  transforms.Resize((240, 320)),
  transforms.ToTensor(),
  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])

data = '/data/imgs/train'

def load_dataset():
  data_path = data
  main_dataset = datasets.ImageFolder(
    root = data_path,
    transform = transform_image
  )

  loader = torch.utils.data.DataLoader(
    dataset = main_dataset,
    batch_size= 64,
    num_workers = 0,
    shuffle= True
  )

  # Dataset has 22424 data points
  trainloader, testloader = random_split(loader.dataset, [21000, 1424])

  return trainloader, testloader

trainloader, testloader = load_dataset()

Now to get a single batch of images from the train and test loaders:

images, labels = next(iter(trainloader))
images.shape
# %%
len(trainloader)

# %%
images_test, labels_test = next(iter(testloader))
images_test.shape

# %%
len(testloader)

The output that I get is does not have the batch size for train or test batches. Teh output dims should be [batch x channel x H x W] but I get [channel x H x W].

Output:

But if I create the split from the dataset and then make two data loaders using the splits, I get the batchsize in the output.

def load_dataset():
    data_path = data
    main_dataset = datasets.ImageFolder(
      root = data_path,
      transform = transform_image
    )
    # Dataset has 22424 data points
    train_data, test_data = random_split(main_dataset, [21000, 1424])

    trainloader = torch.utils.data.DataLoader(
      dataset = train_data,
      batch_size= 64,
      num_workers = 0,
      shuffle= True
    )

    testloader = torch.utils.data.DataLoader(
      dataset = test_data,
      batch_size= 64,
      num_workers= 0,
      shuffle= True
    )

    return trainloader, testloader

trainloader, testloader = load_dataset()

On running the same 4 commands to get a single train and test batch:

Is the first approach wrong? Although the length shows that the data has been split. So why do I not see the batch size?

Output:
The first approach is wrong.

Only DataLoader instances return batches of items. The Dataset like instances don't.

When you call make_split you pass it loader.dataset which is just a reference to main_dataset (not a DataLoader). The result is that trainloader and testloader are Datasets not DataLoaders. In fact you discard loader which is your only DataLoader when you return from load_dataset.

The second version is what you should do to get two separate DataLoaders.


Instruction: Using Torch for numerical optimization, getting optimizer got an empty parameter list,

Input:
I am doing some computations and would like to optimise the parameters of this by using Pytorch. I am NOT defining a neural network, so no layers and stuff like that. Just a simple sequence of computations. I use the torch.nn.Module to be able to use Pytorch's optimisers.

My class looks something like this:

class XTransformer(torch.nn.Module):

    def __init__(self, x):
        super(ReLUTransformer, self).__init__()
        self.x = x

    def funky_function(self, m, c):

        # do some computations
        m = self.x * 2 - m + c

        return m, c

    def forward(self, m, c):
        m, c = self.funky_function(m, c)
        return m, c

Later on I define and try to optimise this parameter x like so:

x = torch.autograd.Variable(x, requires_grad=True)
model = XTransformer(x)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
m, c = smt
loss = loss_func()

for t in range(100):
    m , c = model(m, c)
    l = loss(m, true)
    optimizer.zero_grad()
    l.backward()
    optimizer.step()

I don't know what to do. I get the "ValueError: optimizer got an empty parameter list" error. when I just give [x] as an argument to the optimizer, It doesn't update and change x for me. What should I do?

Output:
You need to register x as a parameter to let PyTorch know this should be a trainable parameter. This can be done by defining it as a nn.Parameter during init

    def __init__(self, x):
        super(ReLUTransformer, self).__init__()
        self.x = torch.nn.Parameter(x)


Instruction: Python list.append(another_list) is just repeating another_list again and again

Input:
I am writing a program which goes through a loop.
In the loop body it make a Python list name value and append this value to another global list values
But i am having a issue that after using

values.append(value)

but it append the value to every element of values

# values
[['closed_eye_0003.jpg_face_2.jpg', 0]]
# value
['closed_eye_0007.jpg_face_1.jpg', 0]
# after appending the value to values the output is
[['closed_eye_0007.jpg_face_1.jpg', 0], ['closed_eye_0007.jpg_face_1.jpg', 0]]

The Code is

import face_recognition
values = list()
value = list()
root_dir = '/content/dataset_facialImages_300/test/CloseFace'
isOpen = 0
for img_name in imgs_names:
  img_file = root_dir + '/' + img_name
  # Load the jpg file into a numpy array
  image = face_recognition.load_image_file(img_file)

  # Find all facial features in all the faces in the image
  face_landmarks_list = face_recognition.face_landmarks(image)
  if len(face_landmarks_list):
    # print(len(face_landmarks_list))
    # print(img_name)
    first_face = face_landmarks_list[0]

    left_eye = first_face['left_eye']
    right_eye = first_face['right_eye']

    value.clear()
    value.append(img_name)
    value.append(isOpen)
    # for i in right_eye:
    #   value.append(i[0])
    #   value.append(i[1])
    print(value)
    # values.insert( len(values), value)
    values.append(value)
    print(values)
  # print( (len(values) / len(imgs_names)) * 100 )

The Output is

['closed_eye_0003.jpg_face_2.jpg', 0]
[['closed_eye_0003.jpg_face_2.jpg', 0]]
['closed_eye_0007.jpg_face_1.jpg', 0]
[['closed_eye_0007.jpg_face_1.jpg', 0], ['closed_eye_0007.jpg_face_1.jpg', 0]]
['closed_eye_0009.jpg_face_1.jpg', 0]
[['closed_eye_0009.jpg_face_1.jpg', 0], ['closed_eye_0009.jpg_face_1.jpg', 0], ['closed_eye_0009.jpg_face_1.jpg', 0]]
['closed_eye_0012.jpg_face_1.jpg', 0]
[['closed_eye_0012.jpg_face_1.jpg', 0], ['closed_eye_0012.jpg_face_1.jpg', 0], ['closed_eye_0012.jpg_face_1.jpg', 0], ['closed_eye_0012.jpg_face_1.jpg', 0]]

If i use

values.extend(value)

it does not repeat

if i use

values.insert( len(values), value)

the output is same

The output should be

['closed_eye_0003.jpg_face_2.jpg', 0]
[['closed_eye_0003.jpg_face_2.jpg', 0]]
['closed_eye_0007.jpg_face_1.jpg', 0]
[['closed_eye_0003.jpg_face_1.jpg', 0], ['closed_eye_0007.jpg_face_1.jpg', 0]]

Output:
Create value = list() inside the loop instead of clearing it.

If you print value and values after clearing value, you would understand what's going on.


Instruction: replacement of nditer for numpy array for pytorch tensor

Input:
I am looking for similar functionality for pytorch tensor as of nditer of numpy array, see this link with a small example.
https://discuss.pytorch.org/t/replacement-of-np-nditer-for-torch/64024?u=songqsh

Output:
I agree with comments that iterating element by element is reraly a good idea for interaction with arrays or tensors. nditer has lots of functionality, here is just an iteration through all elements in the tensor which returns coordinates of the element together with element itself:

def deep_iter(data, ix=tuple()):
    try:
        for i, element in enumerate(data):
            yield from deep_iter(element, ix + (i,))
    except:
        yield ix, data

So for example at pytorch forum it can be used as following:

new_values = {}
for i, value in deep_iter(a):
    if all(map(lambda x: 0 < x < (a.shape[1] - 1), i)):
        new_values[i] = calc_average(i, a) #write func to calc average

for i, new_value in new_values.items():
    a[i] = new_value


Instruction: How to convert list of str into Pytorch tensor

Input:
How do I convert a list of strings into a string/character tensor in Pytorch?

Related example with numpy:

import numpy as np

mylist = ["this","is","my","list"]

np.array([mylist])

Returns:

array([['this', 'is', 'my', 'list']], dtype='<U4')

However, in pytorch:

torch.tensor(mylist)

Returns:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-156-36722d81da09> in <module>
----> 1 torch.tensor(mylist)

ValueError: too many dimensions 'str'

A tensor is a multi-dimensional array, so I'm assuming this is possible pytorch.

Note: this post does not answer my question

Output:
There is no string tensor so you cannot directly convert to pytorch tensor of strings.

Alternative, you can convert the string to ASCII char values and save that as a Tensor.


Instruction: ModuleNotFoundError: No module named 'torch_scope'

Input:
Using the macOS terminal, I'm trying to run ./autoner_train.sh by following this guide on GitHub.

I have activated my Conda environment and check my PyTorch version

(pytorch_env) myname (master) AutoNER $ python -c "import torch; print(torch.__version__)"
1.3.1

After that, when running, I get the following error

  ModuleNotFoundError: No module named 'torch_scope'

I don't know where's the problem. I have installed everything and I tried googling the problem, all I found is that I need PyTorch installed, which I already have.

Output:
I the documentation at the Dependencies section you can read:

  Dependencies

  This project is based on python>=3.6. The dependent package for this
  project is listed as below:

  numpy==1.13.1
  tqdm
  torch-scope>=0.5.0
  pytorch==0.4.1

So you need to install torch-scope>=0.5.0 too:

pip install torch-scope


Instruction: Neural Network for bit operation AND

Input:
This is probably not the best place to ask this question but I am really struggling.
I want to create a "trivial" neural network which has 2 inputs, 3 hidden neurons and 1 output. The idea is that we feed it two booleans and it outputs the AND result.

I wanted to do it in a somewhat "conventional" way (I am still very new to pytorch). I'm using the Adam optimiser with a learning rate of 0.001 and the nll_loss function (although I need to change it as I need to have only 1 output and this requires the number of outputs to match the number of classes - in this case 1 and 0)

I know this should be a really trivial problem but I am really struggling and can't find anything useful on google.

Output:
Make a .csv file with your training data.

x1  x2  label
TRUE    FALSE   FALSE
TRUE    TRUE    TRUE
FALSE   FALSE   FALSE
FALSE   TRUE    FALSE

Create a dataset class.

class boolData(Dataset):
    def __init__(self, csv_path):
        self.label = pd.read_csv(csv_path)

    def __len__(self):
        return len(self.label)

    def __getitem__(self, idx):
        sample = torch.tensor(self.label.iloc[idx,0:2]).int()
        label = torch.tensor(self.label.iloc[idx,2]).int()
        return sample, label

tensor_dataset = boolData(csv_path='sample_bool_stack.csv')

boolDL = DataLoader(tensor_dataset, batch_size=4, shuffle=True)

batch, labels = next(iter(boolDL))

batch, labels

(tensor([[1, 0],
         [1, 1],
         [0, 1],
         [0, 0]], dtype=torch.int32), tensor([0, 1, 0, 0], dtype=torch.int32))

Initialize embeddings.

def _emb_init(x):
    x = x.weight.data
    sc = 2/(x.size(1)+1)
    x.uniform_(-sc,sc)

The model converts True, False to integers and takes the sum as the input.
EDIT: output with just single neuron.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.emb = nn.Embedding(3, 2)
        _emb_init(self.emb)
        self.fc1 = nn.Linear(2, 10)
        self.fc2 = nn.Linear(10, 1)

    def forward(self, x):
        x = torch.sum(x, dim=1)
        x = self.emb(x)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        return x

Initialize model, optimizer, and loss function. EDIT: change loss to match single neuron output.

model = Net()
opt = optim.SGD(model.parameters(), lr=1e-3)
loss_func = nn.BCEWithLogitsLoss()
num_epochs = 10000

device = "cpu"

Train model.

for epoch in range(num_epochs):
    for i,(inputs, labels) in enumerate(boolDL):
        inputs = inputs.to(device).long()
        labels = labels.to(device).float()

        opt.zero_grad()

        output = model(inputs)
        loss = loss_func(output.view(-1), labels)
        loss.backward()
        opt.step()
        with torch.no_grad():
            if epoch % 2000 == 0: print(loss.item())

0.004416308831423521
0.002891995944082737
0.004371378570795059
0.0017852336168289185

Test model.

inputs = torch.tensor([[0,0],[0,1],[0,0]])

def check_model(inputs):
    out = model(inputs)
    preds = torch.sigmoid(out).round()
    return preds.detach().numpy()

check_model(inputs)

array([[0.],
       [0.],
       [0.]], dtype=float32)


Instruction: How to extract value from multi dimensional tensor without losing backward info - PyTorch

Input:
I am trying to extract some specific columns of a multi-dimensional tensor. I can take these values in the concept of multi-dimensional access; however I am losing the "grad_fn" option of the tensor when I create new values.

My actual multi-dimensional tensor, x, is:

tensor([[[-8.5780, -7.1091, -8.9204,  ..., -8.0616, -8.4115, -7.6345],
     [-7.9776, -7.3767, -8.2914,  ..., -7.9634, -9.1003, -7.5687],
     [-7.7192, -7.4307, -8.4294,  ..., -7.8605, -8.1345, -7.0781],
     ...,
     [-8.3652, -7.5910, -8.6671,  ..., -8.0487, -8.5826, -7.8624],
     [-8.1572, -7.1679, -8.8977,  ..., -8.1059, -8.1500, -7.7310],
     [-8.1821, -7.5455, -9.2328,  ..., -8.7435, -8.4430, -7.2503]]],
   grad_fn=<LogSoftmaxBackward>)

with shape (1,10,2000).

For example, I want to extract its specific 10 columns and finally create an array with a shape (1,10,10) or (10,10).

Currently, of course, I can extract these values into an array:

    for i in range(x.size(0)):

        values = np.array([x[i][1:, 1].detach().numpy()])

        idx_to_get = [1,5,7,25,37,44,720,11,25,46]

        for idx in idx_to_get:
            values = np.append(sentence_scores, np.array([np.array(x[0][1:,idx].detach().numpy())]), axis=0)

        values = np.delete(values, 0, axis=0)

        print(torch.from_numpy(values))

Running the code above gave me this output for the values:

tensor([[-7.5589, -6.7990, -7.2068, -7.4451, -7.6688, -7.2991, -7.1398, -7.4362,
     -7.4959, -8.0101, -7.5106, -8.0425, -7.6203, -7.7266, -7.9249, -7.6479,
     -7.6684],
    [-7.2831, -7.7666, -7.8302, -7.3651, -7.2184, -6.7932, -7.1968, -7.6590,
     -7.4033, -6.9504, -7.0767, -7.5366, -7.8364, -7.5935, -8.1235, -7.3222,
     -7.8096],
    [-7.5589, -6.7990, -7.2068, -7.4451, -7.6688, -7.2991, -7.1398, -7.4362,
     -7.4959, -8.0101, -7.5106, -8.0425, -7.6203, -7.7266, -7.9249, -7.6479,
     -7.6684],
    [-7.5650, -7.6627, -7.4230, -7.4726, -7.5621, -7.4489, -7.8344, -7.6130,
     -7.9440, -7.6158, -7.1895, -7.8070, -7.2306, -7.6364, -7.7390, -7.6832,
     -7.5931],
    [-7.5589, -6.7990, -7.2068, -7.4451, -7.6688, -7.2991, -7.1398, -7.4362,
     -7.4959, -8.0101, -7.5106, -8.0425, -7.6203, -7.7266, -7.9249, -7.6479,
     -7.6684],
    [-7.5589, -6.7990, -7.2068, -7.4451, -7.6688, -7.2991, -7.1398, -7.4362,
     -7.4959, -8.0101, -7.5106, -8.0425, -7.6203, -7.7266, -7.9249, -7.6479,
     -7.6684],
    [-7.2831, -7.7666, -7.8302, -7.3651, -7.2184, -6.7932, -7.1968, -7.6590,
     -7.4033, -6.9504, -7.0767, -7.5366, -7.8364, -7.5935, -8.1235, -7.3222,
     -7.8096],
    [-8.3559, -8.3751, -8.2082, -8.6825, -8.4860, -8.4156, -8.4683, -8.8760,
     -8.7354, -8.6155, -8.7544, -8.4527, -8.3690, -8.5795, -8.6023, -8.2267,
     -8.4736],
    [-7.4392, -7.4694, -7.4094, -7.5062, -7.7691, -7.9009, -7.7664, -7.1755,
     -8.0641, -7.6327, -7.6413, -7.9604, -7.9520, -7.8893, -7.8119, -7.8718,
     -8.0961],
    [-8.2182, -8.0280, -8.1398, -8.0258, -7.9951, -8.0664, -8.1976, -7.6182,
     -8.0356, -8.0293, -7.7228, -7.7847, -7.4966, -7.6925, -7.5268, -7.0476,
     -7.2920]])

But the values also should have grad_fn.
What should I do?

I know, using values.requires_grad_(True) worked, but I believe that with using that function, I loss the LogSoftmaxBackward on x.

Output:
The problem is that you can not use numpy functions to get this done AND retain the graph. You must use PyTorch functions only.

x = torch.rand((1,10,2000), requires_grad=True)
idx_to_get = [1,5,7,25,37,44,720,11,25,46]
values = x[0,1:,idx_to_get]
values

tensor([[0.6669, 0.1121, 0.1323, 0.7351, 0.0252, 0.2551, 0.3044, 0.3986, 0.7351,
         0.1060],
        [0.6169, 0.7715, 0.2829, 0.2860, 0.6810, 0.2485, 0.8585, 0.5284, 0.2860,
         0.8363],
        [0.6877, 0.0899, 0.6346, 0.7018, 0.7357, 0.1477, 0.2073, 0.3877, 0.7018,
         0.0226],
        [0.9241, 0.7883, 0.8442, 0.1831, 0.0551, 0.0209, 0.5300, 0.6909, 0.1831,
         0.2950],
        [0.5141, 0.5072, 0.4354, 0.3998, 0.5152, 0.9183, 0.2200, 0.5955, 0.3998,
         0.8631],
        [0.9630, 0.3542, 0.8435, 0.8299, 0.8614, 0.5029, 0.8672, 0.4985, 0.8299,
         0.2207],
        [0.6399, 0.5128, 0.2131, 0.4255, 0.9318, 0.6598, 0.8478, 0.7902, 0.4255,
         0.9080],
        [0.8920, 0.0357, 0.8957, 0.7379, 0.0191, 0.6750, 0.8326, 0.8535, 0.7379,
         0.9087],
        [0.5243, 0.7446, 0.4278, 0.3542, 0.1104, 0.2999, 0.0132, 0.8218, 0.3542,
         0.6761]], grad_fn=<IndexBackward>)


Instruction: Cifar10 dataset: read certain number of images from a class

Input:
I am currently learning deep learning with Pytorch and doing some experiment with Cifar 10 dataset. Which is having 10 classes each class is having 5000 test images. I want to use only 60% of dog and deer classes data and 100% data of other classes.

As per my understanding I need to use custom dataset. But I am not actually able to figure it out. Any Idea, sample code or link if you can share will be helpful for me.

Output:
You can use Subset like this:

from torchvision.datasets import CIFAR10
from torch.utils.data import Subset

ds = CIFAR10('~/.torch/data/', train=True, download=True)
dog_indices, deer_indices, other_indices = [], [], []
dog_idx, deer_idx = ds.class_to_idx['dog'], ds.class_to_idx['deer']

for i in range(len(ds)):
  current_class = ds[i][1]
  if current_class == dog_idx:
    dog_indices.append(i)
  elif current_class == deer_idx:
    deer_indices.append(i)
  else:
    other_indices.append(i)
dog_indices = dog_indices[:int(0.6 * len(dog_indices))]
deer_indices = deer_indices[:int(0.6 * len(deer_indices))]
new_dataset = Subset(ds, dog_indices+deer_indices+other_indices)


Instruction: Constraint on dimensions of activation/feature map in convolutional network

Input:
Let's say input to intermediate CNN layer is of size 512×512×128 and that in the convolutional layer we apply 48 7×7 filters at stride 2 with no padding. I want to know what is the size of the resulting activation map?

I checked some previous posts (e.g., here or here) to point to this Stanford course page. And the formula given there is (W − F + 2P)/S + 1 = (512 - 7)/2 + 1, which would imply that this set up is not possible, as the value we get is not an integer.

However if I run the following snippet in Python 2.7, the code seems to suggest that the size of activation map was computed via (512 - 6)/2, which makes sense but does not match the formula above:

>>> import torch
>>> conv = torch.nn.Conv2d(in_channels=128, out_channels=48, kernel_size=7, stride=2, padding=0)
>>> conv
Conv2d(128, 48, kernel_size=(7, 7), stride=(2, 2))
>>> img = torch.rand((1, 128, 512, 512))
>>> out = conv(img)
>>> out.shape
(1, 48, 253, 253)

Any help in understanding this conundrum is appreciated.

Output:
Here is the formula being used in pytorch: conv2d(go to the shape section)

Also, as far as I know, this is the best tutorial on this subject.

Bonus: here is a neat visualizer for conv calculations.


Instruction: Train part of the network

Input:
I am new to Deep Learning and Pytorch. I have the following problem:

My overall architecture consists of a network that I define (NN1), and another pretrained network (NN2), so that the output of NN1 is the input of NN2. I want to define the loss of NN1 using the difference (RMSE) between the output of NN2 and a know ground-truth.
I need to back-propagate through NN2 and NN1 (to train NN1), without changing NN2.

I can use requires_grad=False on NN2, but will it disable the back-propagation through NN2? How do I specify this requirement in Pytorch?

Thanks.

Output:
You will use two different modules. The first will be your module/model/NN (the NN1) and the second the pretrained one.

Then, you will use model_nn1.train() and model_nn2.eval(). Then, you can do:

optimizer = torch.optim.Adam(model_nn1.parameters())  # You can use your own preferred optimizer

model_nn1.train()
model_nn2.eval()

for epoch in range(epochs):
  for x, y in dataloader:  # the dataloader is your dataloader according to the torch.utils.data.DataLoader
    optimizer.zero_grad()
    h_nn1 = model_nn1(x) # x is the input
    y_hat = model_nn2(h_nn1) # y_hat is the output
    loss = torch.sqrt(torch.mean((yhat-y)**2))
    loss.backward()
    optimizer.step()

You can check the requirement about gradient doing:

>>> import torch
>>> x = torch.nn.Linear(2, 3)
>>> x2 = torch.nn.Linear(3, 2)
>>> z = torch.rand(2, 2)
>>> y = torch.rand(2, 2)
>>> x.train()
Linear(in_features=2, out_features=3, bias=True)
>>> x2.eval()
Linear(in_features=3, out_features=2, bias=True)
>>> h = x(z)
>>> h.requires_grad
True
>>> y_hat = x2(h)
>>> y_hat.requires_grad
True


Instruction: Reading vector information instead when trying to read filename to create custom dataset

Input:
Using python v 3.7.3, pytorch v 0.4.1, windows 10, running code on Jupyter

I am pretty new to programming and deep learning, and creating a new dataset by combining 2 existing datasets.  I am trying to read the filenames of the images but instead it is returning the vector information.  Here is a snippet of the output (imgname is the path, names should just be the filename eg vinegar_41):

imgname: C:\Users\User\Documents\Dataset\train\vinegar\vinegar_41.png

 names:  [[[128 128 128]
  [128 128 128]
  [128 128 128]
  ...
  [128 128 128]
  [128 128 128]
  [128 128 128]]

 [[128 128 128]
  [128 128 128]
  [128 128 128]
  ...
  [128 128 128]
  [128 128 128]
  [128 128 128]]

Here is my code thus far:

__all__ = ['MyDataset']

class MyDataset(Dataset):

    def __init__(self, root, transform=None, target_transform=None):
        root = train_set
        print(root)
        root_list = os.listdir(root)
        print(root_list)
        for f in root_list:
            print('checkpoint')
            imgs = []
            for img in os.listdir(f):
                #print('\n image: ', img)
                imgname = (root + '\\' + f + '\\' + img)
                print('\n imgname:', imgname)
                open(imgname, 'r')
                if img.endswith('.png'):
                    names = cv2.imread(imgname)
                    imgs.append(names)
                    #print('\n imgs: ', imgs)
                    print('\n names: ', names)

                #Image.close()
        self.imgs = imgs
        self.transform = transform
        self.target_transform = target_transform

    def __getitem__(self, index):
        fn, label = self.imgs[index]
        img = PIL.Image.open(fn).convert('RGB')
        if self.transform is not None:
            img = self.transform(img)
        if self.target_transform is not None:
            label = self.transform(label)
        return img, label

    def __len__(self):
        return len(self.imgs)

I'm in over my head, but my project is due in a couple days, so any advice on how to make this work would be greatly appreciated.

Output:
from skimage import io

class MyDataset(Dataset):
    """My dataset."""

    def __init__(self, csv_file, root_dir, transform=None,
                 target_transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with labels.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
            target_transform (callable, optional): Optional transform to
                be applied on a label.
        """
        self.label = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        return len(self.label)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir, self.label.iloc[idx, 0])
        image = io.imread(img_name)
        sample = image
        lab = self.label.iloc[idx, 1]

        if self.transform is not None:
            img = self.transform(img)
        if self.target_transform is not None:
            lab = self.target_transform(lab)

        return sample, lab

The csv file should look like:

fname,label
image1.png,1
image2.png,0
image3.png,1
...
imagen.png,0


Instruction: Cannot deploy trained model to Google Cloud Ai-Platform with custom prediction routine: Model requires more memory than allowed

Input:
I am trying to deploy a pretrained pytorch model to AI Platform with a custom prediction routine. After following the instructions described here the deployment fails with the following error:

ERROR: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error: Model requires more memory than allowed. Please try to decrease the model size and re-deploy. If you continue to have error, please contact Cloud ML.

The contents of the model folder are 83.89 MB large and are below the 250 MB limit that's described in the documentation. The only files in the folder are the checkpoint file (.pth) for the model and the tarball required for the custom prediction routine.

Command to create the model:

gcloud beta ai-platform versions create pose_pytorch --model pose --runtime-version 1.15 --python-version 3.5 --origin gs://rcg-models/pytorch_pose_estimation --package-uris gs://rcg-models/pytorch_pose_estimation/my_custom_code-0.1.tar.gz --prediction-class predictor.MyPredictor

Changing the runtime version to 1.14 leads to the same error.
I have tried changing the --machine-type argument to mls1-c4-m2 like Parth suggested but I still get the same error.

The setup.py file that generates my_custom_code-0.1.tar.gz looks like this:

setup(
    name='my_custom_code',
    version='0.1',
    scripts=['predictor.py'],
    install_requires=["opencv-python", "torch"]
)

Relevant code snippet from the predictor:

    def __init__(self, model):
        """Stores artifacts for prediction. Only initialized via `from_path`.
        """
        self._model = model
        self._client = storage.Client()

    @classmethod
    def from_path(cls, model_dir):
        """Creates an instance of MyPredictor using the given path.

        This loads artifacts that have been copied from your model directory in
        Cloud Storage. MyPredictor uses them during prediction.

        Args:
            model_dir: The local directory that contains the trained Keras
                model and the pickled preprocessor instance. These are copied
                from the Cloud Storage model directory you provide when you
                deploy a version resource.

        Returns:
            An instance of `MyPredictor`.
        """

        net = PoseEstimationWithMobileNet()
        checkpoint_path = os.path.join(model_dir, "checkpoint_iter_370000.pth")
        checkpoint = torch.load(checkpoint_path, map_location='cpu')
        load_state(net, checkpoint)

        return cls(net)

Additionally I have enabled logging for the model in AI Platform and I get the following outputs:

2019-12-17T09:28:06.208537Z OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
2019-12-17T09:28:13.474653Z WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/google/cloud/ml/prediction/frameworks/tf_prediction_lib.py:48: The name tf.saved_model.tag_constants.SERVING is deprecated. Please use tf.saved_model.SERVING instead.
2019-12-17T09:28:13.474680Z {"textPayload":"","insertId":"5df89fad00073e383ced472a","resource":{"type":"cloudml_model_version","labels":{"project_id":"rcg-shopper","region":"","version_id":"lightweight_pose_pytorch","model_id":"pose"}},"timestamp":"2019-12-17T09:28:13.474680Z","logName":"projects/rcg-shopper/logs/ml.googleapis…
2019-12-17T09:28:13.474807Z WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/google/cloud/ml/prediction/frameworks/tf_prediction_lib.py:50: The name tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY is deprecated. Please use tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY instead.
2019-12-17T09:28:13.474829Z {"textPayload":"","insertId":"5df89fad00073ecd4836d6aa","resource":{"type":"cloudml_model_version","labels":{"project_id":"rcg-shopper","region":"","version_id":"lightweight_pose_pytorch","model_id":"pose"}},"timestamp":"2019-12-17T09:28:13.474829Z","logName":"projects/rcg-shopper/logs/ml.googleapis…
2019-12-17T09:28:13.474918Z WARNING:tensorflow:
2019-12-17T09:28:13.474927Z The TensorFlow contrib module will not be included in TensorFlow 2.0.
2019-12-17T09:28:13.474934Z For more information, please see:
2019-12-17T09:28:13.474941Z   * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
2019-12-17T09:28:13.474951Z   * https://github.com/tensorflow/addons
2019-12-17T09:28:13.474958Z   * https://github.com/tensorflow/io (for I/O related ops)
2019-12-17T09:28:13.474964Z If you depend on functionality not listed there, please file an issue.
2019-12-17T09:28:13.474999Z {"textPayload":"","insertId":"5df89fad00073f778735d7c3","resource":{"type":"cloudml_model_version","labels":{"version_id":"lightweight_pose_pytorch","model_id":"pose","project_id":"rcg-shopper","region":""}},"timestamp":"2019-12-17T09:28:13.474999Z","logName":"projects/rcg-shopper/logs/ml.googleapis…
2019-12-17T09:28:15.283483Z ERROR:root:Failed to import GA GRPC module. This is OK if the runtime version is 1.x
2019-12-17T09:28:16.890923Z Copying gs://cml-489210249453-1560169483791188/models/pose/lightweight_pose_pytorch/15316451609316207868/user_code/my_custom_code-0.1.tar.gz...
2019-12-17T09:28:16.891150Z / [0 files][    0.0 B/  8.4 KiB]
2019-12-17T09:28:17.007684Z / [1 files][  8.4 KiB/  8.4 KiB]
2019-12-17T09:28:17.009154Z Operation completed over 1 objects/8.4 KiB.
2019-12-17T09:28:18.953923Z Processing /tmp/custom_code/my_custom_code-0.1.tar.gz
2019-12-17T09:28:19.808897Z Collecting opencv-python
2019-12-17T09:28:19.868579Z   Downloading https://files.pythonhosted.org/packages/d8/38/60de02a4c9013b14478a3f681a62e003c7489d207160a4d7df8705a682e7/opencv_python-4.1.2.30-cp37-cp37m-manylinux1_x86_64.whl (28.3MB)
2019-12-17T09:28:21.537989Z Collecting torch
2019-12-17T09:28:21.552871Z   Downloading https://files.pythonhosted.org/packages/f9/34/2107f342d4493b7107a600ee16005b2870b5a0a5a165bdf5c5e7168a16a6/torch-1.3.1-cp37-cp37m-manylinux1_x86_64.whl (734.6MB)
2019-12-17T09:28:52.401619Z Collecting numpy>=1.14.5
2019-12-17T09:28:52.412714Z   Downloading https://files.pythonhosted.org/packages/9b/af/4fc72f9d38e43b092e91e5b8cb9956d25b2e3ff8c75aed95df5569e4734e/numpy-1.17.4-cp37-cp37m-manylinux1_x86_64.whl (20.0MB)
2019-12-17T09:28:53.550662Z Building wheels for collected packages: my-custom-code
2019-12-17T09:28:53.550689Z   Building wheel for my-custom-code (setup.py): started
2019-12-17T09:28:54.212558Z   Building wheel for my-custom-code (setup.py): finished with status 'done'
2019-12-17T09:28:54.215365Z   Created wheel for my-custom-code: filename=my_custom_code-0.1-cp37-none-any.whl size=7791 sha256=fd9ecd472a6a24335fd24abe930a4e7d909e04bdc4cf770989143d92e7023f77
2019-12-17T09:28:54.215482Z   Stored in directory: /tmp/pip-ephem-wheel-cache-i7sb0bmb/wheels/0d/6e/ba/bbee16521304fc5b017fa014665b9cae28da7943275a3e4b89
2019-12-17T09:28:54.222017Z Successfully built my-custom-code
2019-12-17T09:28:54.650218Z Installing collected packages: numpy, opencv-python, torch, my-custom-code

Output:
This is a common problem and we understand this is a pain point. Please do the following:

torchvision has torch as dependency and by default, it pulls torch from pypi.

When deploying the model, even if you point to use custom ai-platform torchvision packages it will do it, since torchvision when is built by PyTorch team, it is configured to use torch as dependency. This torch dependency from pypi, gives a 720mb file because it includes the GPU units

To solve #1, you need to build torchvision from source and tell torchvision where you want to get torch from, you need to set it to go to the torch website as the package is smaller. Rebuild the torchvision binary using Python PEP-0440 direct references feature. In torchvision setup.py we have:

pytorch_dep = 'torch'
if os.getenv('PYTORCH_VERSION'):
    pytorch_dep += "==" + os.getenv('PYTORCH_VERSION')

Update setup.py in torchvision to use direct references feature:

requirements = [
     #'numpy',
     #'six',
     #pytorch_dep,
     'torch @ https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl'
]

* I already did this for you*,  so I build 3 wheel files you can use:

gs://dpe-sandbox/torchvision-0.4.0-cp37-cp37m-linux_x86_64.whl (torch 1.2.0, vision 0.4.0)
gs://dpe-sandbox/torchvision-0.4.2-cp37-cp37m-linux_x86_64.whl (torch 1.2.0, vision 0.4.2)
gs://dpe-sandbox/torchvision-0.5.0-cp37-cp37m-linux_x86_64.whl (torch 1.4.0  vision 0.5.0)

These torchvision packages will get torch from the torch site instead of pypi: (Example: https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl)

Update your model setup.py when deploying the model to AI Platform so it does not include torch nor torchvision.
Redeploy the model as follows:

PYTORCH_VISION_PACKAGE=gs://dpe-sandbox/torchvision-0.5.0-cp37-cp37m-linux_x86_64.whl

gcloud beta ai-platform versions create {MODEL_VERSION} --model={MODEL_NAME} \
            --origin=gs://{BUCKET}/{GCS_MODEL_DIR} \
            --python-version=3.7 \
            --runtime-version={RUNTIME_VERSION} \
            --machine-type=mls1-c4-m4 \
            --package-uris=gs://{BUCKET}/{GCS_PACKAGE_URI},{PYTORCH_VISION_PACKAGE}\
            --prediction-class={MODEL_CLASS}

You can change the PYTORCH_VISION_PACKAGE to any of the options I mentioned in #2


Instruction: Pytorch does not backpropagate through a iterative tensor construction

Input:
I am currently trying to build a tensor iteratively in Pytorch.
Sadly the backprop does not work with the inplace operation in the loop. I already tried equivalent programs with stack for example. Does somebody know how I could build the tensor with a working backprop?

This is a minimal example which produces the error:

import torch

k=2
a =torch.Tensor([10,20])
a.requires_grad_(True)
b = torch.Tensor([10,20])
b.requires_grad_(True)

batch_size = a.size()[0]
uniform_samples = Uniform(torch.tensor([0.0]), torch.tensor([1.0])).rsample(torch.tensor([batch_size,k])).view(-1,k)
exp_a = 1/a
exp_b = 1/b
km = (1- uniform_samples.pow(exp_b)).pow(exp_a)

sticks = torch.zeros(batch_size,k)
remaining_sticks = torch.ones_like(km[:,0])
for i in range(0,k-1):
    sticks[:,i] = remaining_sticks * km[:,i]
    remaining_sticks *= (1-km[:,i])
sticks[:,k-1] = remaining_sticks
latent_variables = sticks

latent_variables.sum().backward()

The stack trace:

/opt/conda/conda-bld/pytorch_1570910687230/work/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 563, in start
    self.io_loop.start()
  File "/opt/conda/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 148, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
    handle._run()
  File "/opt/conda/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/opt/conda/lib/python3.6/site-packages/tornado/ioloop.py", line 690, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/opt/conda/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 787, in inner
    self.run()
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 361, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 268, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 541, in execute_request
    user_expressions, allow_stdin,
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 300, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2855, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in _run_cell
    return runner(coro)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 68, in _pseudo_sync_runner
    coro.send(None)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3058, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3249, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-124-2bbdbc3af797>", line 16, in <module>
    sticks[:,i] = remaining_sticks * km[:,i]

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-124-2bbdbc3af797> in <module>
     19 latent_variables = sticks
     20
---> 21 latent_variables.sum().backward()

/opt/conda/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    148                 products. Defaults to ``False``.
    149         """
--> 150         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    151
    152     def register_hook(self, hook):

/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---> 99         allow_unreachable=True)  # allow_unreachable flag
    100
    101

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Output:
You can not do any inplace operations. So you may not use *= in your algorithm.

k = 2
a = torch.tensor(np.array([10.,20]), requires_grad=True).float()
b = torch.tensor(np.array([10.,20]), requires_grad=True).float()

batch_size = a.size()[0]
uniform_samples = Uniform(torch.tensor([0.]), torch.tensor([1.])).rsample(torch.tensor([batch_size,k])).view(-1,k)
exp_a = 1/a
exp_b = 1/b
km = (1 - uniform_samples**exp_b)**exp_a

sticks = torch.zeros(batch_size,k)
remaining_sticks = torch.ones_like(km[:,0])
for i in range(0,k-1):
    sticks[:,i] = remaining_sticks * km[:,i]
    remaining_sticks = remaining_sticks * (1-km[:,i])
sticks[:,k-1] = remaining_sticks
latent_variables = sticks
latent_variables = torch.sum(latent_variables)

latent_variables.backward()


Instruction: LSTM in Pytorch: how to add/change sequence length dimension?

Input:
I am running LSTM in pytorch but as I understand, it is only taking sequence length = 1. When I reshape to have sequence length to 4 or other number, then I get an error of mismatching length in input and target. If I reshape both input and target, then the model complains that it does not accept multi-target labels.

My train dataset has 66512 rows and 16839 columns, 3 categories/classes in the target. I would like to use a batch size 200 and a sequence length of 4, i.e. use 4 rows of data in a sequence.

Please advise how to adjust my model and/or data to be able to run model for various sequence lengths (e.g., 4).

batch_size=200
import torch
from torch.utils.data import TensorDataset
from torch.utils.data import DataLoader
train_target = torch.tensor(train_data[['Label1','Label2','Label3']].values.astype(np.float32))
train_target = np.argmax(train_target, axis=1)
train = torch.tensor(train_data.drop(['Label1','Label2','Label3'], axis = 1).values.astype(np.float32))
train_tensor = TensorDataset(train.unsqueeze(1), train_target)
train_loader = DataLoader(dataset = train_tensor, batch_size = batch_size, shuffle = True)

print(train.shape)
print(train_target.shape)

torch.Size([66512, 16839])
torch.Size([66512])

import torch.nn as nn

class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(LSTMModel, self).__init__()
        # Hidden dimensions
        self.hidden_dim = hidden_dim

        # Number of hidden layers
        self.layer_dim = layer_dim

        # Building LSTM
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)

        # Readout layer
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):

        # Initialize hidden state with zeros
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_().to(device)

        # Initialize cell state
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_().to(device)

        out, (hn, cn) = self.lstm(x, (h0,c0))

        # Index hidden state of last time step
        out = self.fc(out[:, -1, :])

        return out

input_dim = 16839
hidden_dim = 100
output_dim = 3
layer_dim = 1

batch_size = batch_size
num_epochs = 1

model = LSTMModel(input_dim, hidden_dim, layer_dim, output_dim)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model.to(device)

criterion = nn.CrossEntropyLoss()
learning_rate = 0.1

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

print(len(list(model.parameters())))
for i in range(len(list(model.parameters()))):
    print(list(model.parameters())[i].size())

6
torch.Size([400, 16839])
torch.Size([400, 100])
torch.Size([400])
torch.Size([400])
torch.Size([3, 100])
torch.Size([3])

for epoch in range(num_epochs):
    for i, (train, train_target) in enumerate(train_loader):
        # Load data as a torch tensor with gradient accumulation abilities
        train = train.requires_grad_().to(device)
        train_target = train_target.to(device)

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(train)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, train_target)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()
print('Epoch: {}. Loss: {}. Accuracy: {}'.format(epoch, np.around(loss.item(), 4), np.around(accuracy,4)))

Output:
This is what worked eventually - reshaping the input data into sequences of 4 and having one target value per sequence, for which I picked last value in the target sequence per my problem logic. Seems very easy now but was very tricky back then. The rest of the posted code is the same.

train_target = torch.tensor(train_data[['Label1','Label2','Label3']].iloc[3::4].values.astype(np.float32))
train_target = np.argmax(train_target, axis=1)
train = torch.tensor(train_data.drop(['Label1','Label2','Label3'], axis = 1).values.reshape(-1, 4, 16839).astype(np.float32))
train_tensor = TensorDataset(train, train_target)
train_loader = DataLoader(dataset = train_tensor, batch_size = batch_size, shuffle = True)

print(train.shape)
print(train_target.shape)

torch.Size([16628, 4, 16839])
torch.Size([16628])


Instruction: Pytorch: How to access CrossEntropyLoss() gradient?

Input:
I want to modify the tensor that stores the CrossEntropyLoss() gradient, that is, P(i)-T(i). Where is it stored and how do I access it?

code:

input = torch.randn(3, 5, requires_grad=True)
input.register_hook(lambda x: print(" \n input hook: ",x))
print(input)
target = torch.empty(3, dtype=torch.long).random_(5)
print(target)

criterion = nn.CrossEntropyLoss()
criterion.requires_grad = True
loss0 = criterion(input,target)
loss0.register_hook(lambda x: print(" \n loss0 hook: ",x))
print("before backward loss0.grad :",loss0.grad)
print("loss0 :",loss0)
loss0.backward()
print("after backward loss0.grad :",loss0.grad)

output:

tensor([[-0.6149, -0.8179,  0.6084, -0.2837, -0.5316],
        [ 1.7246,  0.5348,  1.3646, -0.7148, -0.3421],
        [-0.3478, -0.6732, -0.7610, -1.0381, -0.5570]], requires_grad=True)
tensor([4, 1, 0])
before backward loss0.grad : None
loss0 : tensor(1.7500, grad_fn=<NllLossBackward>)

 loss0 hook:  tensor(1.)

 input hook:  tensor([[ 0.0433,  0.0354,  0.1472,  0.0603, -0.2862],
        [ 0.1504, -0.2876,  0.1050,  0.0131,  0.0190],
        [-0.2432,  0.0651,  0.0597,  0.0452,  0.0732]])
after backward loss0.grad : None

Output:
Given your specification in the comments you want the gradient with respect to the input (output of the model), in your code you look at the gradient of the loss which does not exist. So you could so something like:

import torch
input = torch.tensor([1,0,1,0], dtype=float, requires_grad=True)
target = torch.tensor([1,2,3,4], dtype=float)
loss = (input - target).abs().mean()
loss.backward()

Here loss.grad gives you None, but input.grad returns:

tensor([ 0.0000, -0.2500, -0.2500, -0.2500], dtype=torch.float64)

Which should be the gradient you are interested in.


Instruction: How do i save images for each class using Pytorch? (no grid)

Input:
I am using acgan to do image augmentation. At present, sample images are generated in grid format. But I want to save images for each class separately. (e.g. 1.png; 2.png ...) How should I modify this code? Or is there an answer that I would like to refer to?

class Generator(nn.Module):
def __init__(self):
    super(Generator, self).__init__()

    self.label_emb = nn.Embedding(opt.n_classes, opt.latent_dim)

    self.init_size = opt.img_size // 4  # Initial size before upsampling
    self.l1 = nn.Sequential(nn.Linear(opt.latent_dim, 128 * self.init_size ** 2))

    self.conv_blocks = nn.Sequential(
        nn.BatchNorm2d(128),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 128, 3, stride=1, padding=1),
        nn.BatchNorm2d(128, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 64, 3, stride=1, padding=1),
        nn.BatchNorm2d(64, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Conv2d(64, opt.channels, 3, stride=1, padding=1),
        nn.Tanh(),
    )

..

generator = Generator()

..

def sample_image(n_row, batches_done):
   """Saves a grid of generated digits ranging from 0 to n_classes"""
   # Sample noise
   z = Variable(FloatTensor(np.random.normal(0, 1, (n_row ** 2, opt.latent_dim))))
   # Get labels ranging from 0 to n_classes for n rows
   labels = np.array([num for _ in range(n_row) for num in range(n_row)])
   labels = Variable(LongTensor(labels))
   gen_imgs = generator(z, labels)
   save_image(gen_imgs.data, "images/%d.png" % batches_done, nrow=n_row, normalize=True)

Output:
in the def sample_image, you have line that defines target labels for the generator:
labels = np.array([num for _ in range(n_row) for num in range(n_row)]).

Instead of using num which changes due to being sampled from range, use constant number you pass as an argument (class_id below):

def sample_image(n_row, batches_done, class_id):
   """Saves a grid of generated digits ranging from 0 to n_classes"""
   # Sample noise
   z = Variable(FloatTensor(np.random.normal(0, 1, (n_row ** 2, opt.latent_dim))))
   # Get labels ranging from 0 to n_classes for n rows
   labels = np.array([class_id for _ in range(n_row) for __ in range(n_row)])
   labels = Variable(LongTensor(labels))
   gen_imgs = generator(z, labels)
   save_image(gen_imgs.data, "images/%d.png" % batches_done, nrow=n_row, normalize=True)

This way you will get a rectangular array full of images of the class you have requested.

Furthermore, to have just one image you can set n_row to 1. Note that you didn't provide code for save_image function, there may be some tricks to it.


Instruction: nvprof warning on CUDA_VISIBLE_DEVICES

Input:
When I use os.environ['CUDA_VISIBLE_DEVICES'] in pytorch, I get the following message

Warning: Device on which events/metrics are configured are different than the device on which it is being profiled. One of the possible reason is setting CUDA_VISIBLE_DEVICES inside the application.

What does this actually mean? How can I avoid this by using 'CUDA_VISIBLE_DEVICES' (not torch.cuda.set_device())?

Here is the code in pytorch test.py

import torch
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '1'
g = 1
c1 = 512
c2 = 512
input = torch.randn(64, c1, 28, 28).cuda()
model = nn.Sequential(
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
    ).cuda()
out = model(input)

and the command:

nvprof --analysis-metrics -o metrics python test.py

Output:
  What does this actually mean?

It means that nvprof started profiling your code on a GPU context which you made unavailable by setting CUDA_VISIBLE_DEVICES.

  How can I avoid this by using CUDA_VISIBLE_DEVICES (not torch.cuda.set_device())?

Probably like this:

import os
os.environ['CUDA_VISIBLE_DEVICES'] = '1'

import torch
....

I know nothing about pytorch, but I would guess that importing the library triggers a lot of CUDA activity you don't see. If you import the library after you set CUDA_VISIBLE_DEVICES, I suspect the whole problem will disappear.

If that doesn't work then you would have no choice but to not set CUDA_VISIBLE_DEVICES within the python code at all, and instead do this:

CUDA_VISIBLE_DEVICES=1 nvprof --analysis-metrics -o metrics python test.py


Instruction: Resolution preserving Fully Convolutional Network

Input:
I am new to ML and Pytorch and I have the following problem:

I am looking for a Fully Convolutional Network architecture in Pytorch, so that the input would be an RGB image (HxWxC or 480x640x3) and the output would be a single channel image (HxW or 480x640). In other words, I am looking for a network that will preserve the resolution of the input (HxW), and will loose the channel dimension. All of the networks that I've came across (ResNet, Densenet, ...) end with a fully connected layer (without any upsampling or deconvolution). This is problematic for two reasons:

I am restricted with the choice of the input size (HxWxC).
It has nothing to do with the output that I expect to get (a single channel image HxW).

What am I missing? Why is there even a FC layer? Why is there no up-sampling, or  some deconvolution layers after feature extraction? Is there any build-in torchvision.model that might suit my requirements? Where can I find such pytorch architecture? As I said, I am new in this field so I don't really like the idea of building such a network from scratch.

Thanks.

Output:
You probably came across the networks that are used in classification. So they end up with a pooling and a fully connected layer to produce a fixed number of categorical output.

Have a look at Unet
https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/

Note: the original unet implementation use a lot of tricks.
You can simply downsample and then upsample symmetrically to do the work.


Instruction: Matrix Multiplication in 3,4 axes pytorch

Input: I've two tensors of shape a(16,8,8,64) and b(64,64). Suppose, I extract last dimension of ainto another column vector c, I want to compute matmul(matmul(c.T, b), c). I want this to be done in each of the first 3 dimensions of a. That is the final product should be of shape (16,8,8,1). How can I achieve this in pytorch?

Output:
Can be done as follows:

row_vec = a[:, :, :, None, :].float()
col_vec = a[:, :, :, :, None].float()
b = (b[None, None, None, :, :]).float()
prod = torch.matmul(torch.matmul(row_vec, b), col_vec)


Instruction: Get probability of multi-token word in MASK position

Input:
It is relatively easy to get a token's probability according to a language model, as the snippet below shows. You can get the output of a model, restrict yourself to the output of the masked token, and then find the probability of your requested token in the output vector. However, this only works with single-token words, e.g. words that are themselves in the tokenizer's vocabulary. When a word does not exist in the vocabulary, the tokenizer will chunk it up into pieces that it does know (see the bottom of the example). But since the input sentence consists of only one masked position, and the requested token has more tokens than that, how can we get its probability? Ultimately I am looking for a solution that works regardless of the number of subword units a word has.

In the code below I have added many comments explaining what is going on, as well as printing out the given output of print statements. You'll see that predicting tokens such as 'love' and 'hate' is straightforward because they are in the tokenizer's vocabulary. 'reprimand' is not, though, so it cannot be predicted in a single masked position - it consists of three subword units. So how can we predict 'reprimand' in the masked position?

from transformers import BertTokenizer, BertForMaskedLM
import torch

# init model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()
# init softmax to get probabilities later on
sm = torch.nn.Softmax(dim=0)
torch.set_grad_enabled(False)

# set sentence with MASK token, convert to token_ids
sentence = f"I {tokenizer.mask_token} you"
token_ids = tokenizer.encode(sentence, return_tensors='pt')
print(token_ids)
# tensor([[ 101, 1045,  103, 2017,  102]])
# get the position of the masked token
masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero().item()

# forward
output = model(token_ids)
last_hidden_state = output[0].squeeze(0)
# only get output for masked token
# output is the size of the vocabulary
mask_hidden_state = last_hidden_state[masked_position]
# convert to probabilities (softmax)
# giving a probability for each item in the vocabulary
probs = sm(mask_hidden_state)

# get probability of token 'hate'
hate_id = tokenizer.convert_tokens_to_ids('hate')
print('hate probability', probs[hate_id].item())
# hate probability 0.008057191967964172

# get probability of token 'love'
love_id = tokenizer.convert_tokens_to_ids('love')
print('love probability', probs[love_id].item())
# love probability 0.6704086065292358

# get probability of token 'reprimand' (?)
reprimand_id = tokenizer.convert_tokens_to_ids('reprimand')
# reprimand is not in the vocabulary, so it needs to be split into subword units
print(tokenizer.convert_ids_to_tokens(reprimand_id))
# [UNK]

reprimand_id = tokenizer.encode('reprimand', add_special_tokens=False)
print(tokenizer.convert_ids_to_tokens(reprimand_id))
# ['rep', '##rim', '##and']
# but how do we now get the probability of a multi-token word in a single-token position?

Output:
Since the split word does not present in the dictionary, BERT is simply unaware of it's probability, so there is no use of masking it before tokenization.

And you can't get it's probability by exploiting rule of chain, see response by J.Devlin. To illustrate it, let's take more generic example. Try to estimate the probability of some bigram in position i. While you can estimate probability of each word given the sentence and their positions

P(w_i|w_0, w_1... w_i-1, w_i+1, ..., w_N),

P(w_i+1|w_0, w_1... w_i, wi+2, ..., w_N),

there is no way to get the probability of the bigram

P(w_i,w_i+1|w_0, w_1... w_i-1, wi+2, ..., w_N)

because BERT does not store such information.

Having said all that, you can get a very rough estimate of the probability of your OOV word by multiplying probabilities of seeing it's parts. So you will get

P("reprimand"|...) ~= P("rep"|...)*P("##rim"|...)*P("##and"|...)

Since your subwords are not regular words, but a special kind of words, this is not all wrong, because the dependency between them is implicit.


Instruction: Calculate mean across one specific dimension of a 4D tensor in Pytorch

Input:
I have a PyTorch video feature tensor of shape [66,7,7,1024] and I need to convert it to [1024,66,7,7]. How to rearrange a tensor shape? Also, how to perform mean across dimension=1? i.e., after performing mean of the dimension with size 66, I need the tensor to be [1024,1,7,7].

I have tried to calculate the mean of dimension=1 but I failed to replace it with the mean value. And I could not imagine a 4D tensor in which one dimension is replaced by its mean.

Edit:
 I tried torch.mean(my_tensor, dim=1). But this returns me a tensor of shape [1024,7,7]. The 4D tensor is being converted to 3D. But I want it to remain 4D with shape [1024,1,7,7].

Thank you very much.

Output:
The first part of the question has been answered in the comments section. So we can use tensor.transpose([3,0,1,2]) to convert the tensor to the shape [1024,66,7,7].

Now mean over the temporal dimension can be taken by
torch.mean(my_tensor, dim=1)
This will give a 3D tensor of shape [1024,7,7].
To obtain a tensor of shape [1024,1,7,7], I had to unsqueeze in dimension=1:

tensor = tensor.unsqueeze(1)


Instruction: converting 1 x 1 x 33 tensor to 1 x 1 x 34 while maintaining values and leaving the 34th index 0

Input: I have a tensor of 0 and and one 1 in a 1 by 1 by 33 tensor and I want all the indexes that are 1 to stay one, but be in a 1 by 1 by 34 tensor. What's the standard practice for this? Apparently using "reshape is bad" for this.

Output:
You can use pad to pad either the beginning or end of a tensor with zeros.

For example

import torch.nn.functional as F
x = F.pad(x, (0, 1))

will pad the end of the last dimension of a 3d tensor x with 1 zero.


Instruction: Python multiprocessing.pool's interaction with a class objective function and neuro-evolution

Input:
Warning, this is gonna be long since I want to be as specific as I can be.

Exact problem: This is a multi-processing  problem. I have ensured that my classes all behave as built/expected in previous experiments.

edit: said threading beforehand.

When I run toy example of my problem in a threaded environment, everything behaves; however, when I transition into my real problem, the code breaks. Specifically, I get a TypeError: can't pickle _thread.lock objects error. Full stack is at the bottom.

My threading needs here are bit different than the example I adapted my code from -- https://github.com/CMA-ES/pycma/issues/31. In this example we have one fitness function that can be independently called by each evaluation and none of the function calls can interact with each other. However, in my real problem we are trying to optimize neural network weights using a genetic algorithm. The GA will suggest potential weights and we need to evaluate these NN controller-weights in our environment. In a single threaded case, we can have just one environment where we evaluate the weights with a simple for-loop: [nn.evaluate(weights) for weights in potential_candidates], find the best-performing individual, and use those weights in the next mutation round. However, we cannot simply have one simulation in a threaded environment.

So, instead of passing in a single function to evaluate I am passing in a list of function (one for each individual, where the environment is the same, but we have forked the processes so that the communication streams don't interact between individuals.)

One further thing of immediate note:
I am using a build-for-parallel evaluation data-structure from neat

from neat.parallel import ParallelEvaluator  # uses multiprocessing.Pool

Toy example code:

NPARAMS = nn.flat_init_weights.shape[0]    # make this a 1000-dimensional problem.
NPOPULATION = 5                            # use population size of 5.
MAX_ITERATION = 100                        # run each solver for 100 function calls.

import time
from neat.parallel import ParallelEvaluator  # uses multiprocessing.Pool
import cma

def fitness(x):
    time.sleep(0.1)
    return sum(x**2)

# # serial evaluation of all solutions
# def serial_evals(X, f=fitness, args=()):
#     return [f(x, *args) for x in X]

# parallel evaluation of all solutions
def _evaluate2(self, weights, *args):
    """redefine evaluate without the dependencies on neat-internal data structures
    """
    jobs = []
    for i, w in enumerate(weights):
        jobs.append(self.pool.apply_async(self.eval_function[i], (w, ) + args))

    return [job.get() for job in jobs]

ParallelEvaluator.evaluate2 = _evaluate2
parallel_eval = ParallelEvaluator(12, [fitness]*NPOPULATION)

# time both
for eval_all in [parallel_eval.evaluate2]:
    es = cma.CMAEvolutionStrategy(NPARAMS * [1], 1, {'maxiter': MAX_ITERATION,
                                                     'popsize': NPOPULATION})
    es.disp_annotation()
    while not es.stop():
        X = es.ask()
        es.tell(X, eval_all(X))
    es.disp()

Necessary background:

When I switch from the toy example to my real code, the above fails.

My classes are:

LevelGenerator (simple GA class that implements mutate, etc)
GridGame (OpenAI wrapper; launches a Java server in which to run the simulation;
          handles all communication between the Agent and the environment)
Agent    (neural-network class, has an evaluate fn which uses the NN to play a single rollout)
Objective (handles serializing/de-serializing weights: numpy <--> torch; launching the evaluate function)

# The classes get composed to get the necessary behavior:
env   = GridGame(Generator)
agent = NNAgent(env)                # NNAgent is a subclass of (Random) Agent)
obj   = PyTorchObjective(agent)

# My code normally all interacts like this in the single-threaded case:

def test_solver(solver): # Solver: CMA-ES, Differential Evolution, EvolutionStrategy, etc
    history = []
    for j in range(MAX_ITERATION):
        solutions = solver.ask() #2d-numpy array. (POPSIZE x NPARAMS)
        fitness_list = np.zeros(solver.popsize)
        for i in range(solver.popsize):
            fitness_list[i] = obj.function(solutions[i], len(solutions[i]))
        solver.tell(fitness_list)
        result = solver.result() # first element is the best solution, second element is the best fitness
        history.append(result[1])

        scores[j] = fitness_list

    return history, result

So, when I attempt to run:

NPARAMS = nn.flat_init_weights.shape[0]
NPOPULATION = 5
MAX_ITERATION = 100

_x = NNAgent(GridGame(Generator))

gyms = [_x.mutate(0.0) for _ in range(NPOPULATION)]
objs = [PyTorchObjective(a) for a in gyms]

def evaluate(objective, weights):
    return objective.fun(weights, len(weights))

import time
from neat.parallel import ParallelEvaluator  # uses multiprocessing.Pool
import cma

def fitness(agent):
    return agent.evalute()

# # serial evaluation of all solutions
# def serial_evals(X, f=fitness, args=()):
#     return [f(x, *args) for x in X]

# parallel evaluation of all solutions
def _evaluate2(self, X, *args):
    """redefine evaluate without the dependencies on neat-internal data structures
    """
    jobs = []
    for i, x in enumerate(X):
        jobs.append(self.pool.apply_async(self.eval_function[i], (x, ) + args))

    return [job.get() for job in jobs]

ParallelEvaluator.evaluate2 = _evaluate2
parallel_eval = ParallelEvaluator(12, [obj.fun for obj in objs])
# obj.fun takes in the candidate weights, loads them into the NN, and then evaluates the NN in the environment.

# time both
for eval_all in [parallel_eval.evaluate2]:
    es = cma.CMAEvolutionStrategy(NPARAMS * [1], 1, {'maxiter': MAX_ITERATION,
                                                     'popsize': NPOPULATION})
    es.disp_annotation()
    while not es.stop():
        X = es.ask()
        es.tell(X, eval_all(X, NPARAMS))
    es.disp()

I get the following error:

TypeError                            Traceback (most recent call last)
<ipython-input-57-3e6b7bf6f83a> in <module>
      6     while not es.stop():
      7         X = es.ask()
----> 8         es.tell(X, eval_all(X, NPARAMS))
      9     es.disp()

<ipython-input-55-2182743d6306> in _evaluate2(self, X, *args)
     14         jobs.append(self.pool.apply_async(self.eval_function[i], (x, ) + args))
     15
---> 16     return [job.get() for job in jobs]

<ipython-input-55-2182743d6306> in <listcomp>(.0)
     14         jobs.append(self.pool.apply_async(self.eval_function[i], (x, ) + args))
     15
---> 16     return [job.get() for job in jobs]

~/miniconda3/envs/thesis/lib/python3.7/multiprocessing/pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658
    659     def _set(self, i, obj):

~/miniconda3/envs/thesis/lib/python3.7/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
    429                         break
    430                     try:
--> 431                         put(task)
    432                     except Exception as e:
    433                         job, idx = task[:2]

~/miniconda3/envs/thesis/lib/python3.7/multiprocessing/connection.py in send(self, obj)
    204         self._check_closed()
    205         self._check_writable()
--> 206         self._send_bytes(_ForkingPickler.dumps(obj))
    207
    208     def recv_bytes(self, maxlength=None):

~/miniconda3/envs/thesis/lib/python3.7/multiprocessing/reduction.py in dumps(cls, obj, protocol)
     49     def dumps(cls, obj, protocol=None):
     50         buf = io.BytesIO()
---> 51         cls(buf, protocol).dump(obj)
     52         return buf.getbuffer()
     53

TypeError: can't pickle _thread.lock objects

I also read here that this might be being caused by the fact that this is a class function -- TypeError: can't pickle _thread.lock objects -- so I created the global scoped fitness function def fitness(agent): return agent.evalute(), but that didn't work either.

I thought this error might be coming from the fact that originally, I had the evaluate function in the PyTorchObjective class as a lambda function, but when I changed that it still broke.

Any insight would be greatly appreciated, and thanks for reading this giant wall of text.

Output:
You are not using multiple threads. You are using multiple processes.

All arguments that you pass to apply_async, including the function itself, are serialized (pickled) under the hood and passed to a worker process via an IPC channel (read up multiprocessing documentation for details). So you cannot pass any entities that are tied to things that are by their nature process-local. This includes  most synchronization primitives since they have to use locks to do atomic operations.

Whenever this happens (as many other questions on this error message show), you are likely trying to be too smart and passing to a parallelization framework an object that already has parallelization logic built in.

If you want to create "multiple levels of parallelization" with such "parallelized object", you'll be better off either:

using the parallelization mechanism of that object proper and not bother about multiple levels: you can't do more stuff at a time than you have cores anyway; or
create and use these "parallelized objects" inside worker processes

but you are likely to hit multiprocessing limitations here since its worker processes are deliberately prohibited from spawning their own pools.

You can let workers add extra items to the work queue but may hit Queue limitations as well.

so for such a scenario, a more advanced 3rd-party distributed work queue solution may be preferrable.


Instruction: is loaded model does not need eval?

Input:
tryint to use the test after I load the model

net = net.load_state_dict(torch.load(PATH))
net.eval()

but this spit the error

net.eval() AttributeError: '_IncompatibleKeys' object has no attribute 'eval'

Output:
here you don't need to assign net.load_state_dict to net

net = net.load_state_dict(torch.load(PATH))

just used

net.load_state_dict(torch.load(PATH))
net.eval()

more see Recommended approach for saving a model


Instruction: Interpolate one range of numbers to another in a tensor in PyTorch

Input:
I am trying to transform a set of numbers say {0, 4, 6} to a consecutive one {0, 1, 2}.
Currently I am using Scipy's interp1d function. My problem is that I want to change each element of a tensor. Is there a better way than broadcasting the interp1d function to each element in a tensor? If not how can I achieve this broadcasting of a custom function

Currently I am doing the following:

>>> a = torch.tensor([[0, 6, 4],
                      [4, 0, 4]])
>>> unique_values = np.unique(a)
>>> interpolator = interp1d(unique_values, range(len(unique_values)))
>>> interpolator(a[0, 1])
>>> 2

Output:
If you just want to replace, why do you want to interpolate?

a[a==6]=2
a[a==4]=1

Should work. If you don't want to hardcode, you can find unique values like you did, sort them and then replace them in a loop.


Instruction: PyTorch - How to derive manually ResNet?

Input:
I managed to find this implementation of ResNet-18:

def __init__(self):
    super(ResNet18,self).__init__()
    self.block1 = nn.Sequential(
        nn.Conv2d(1,64,kernel_size=2,stride=2,padding=3,bias=False),
        nn.BatchNorm2d(64),
        nn.ReLU(True)
    )
    self.block2 = nn.Sequential(
       nn.MaxPool2d(1,1),
       ResidualBlock(64,64),
       ResidualBlock(64,64,2)
    )
    self.block3 = nn.Sequential(
        ResidualBlock(64,128),
        ResidualBlock(128,128,2)
    )
    self.block4 = nn.Sequential(
        ResidualBlock(128,256),
        ResidualBlock(256,256,2)
    )
    self.block5 = nn.Sequential(
        ResidualBlock(256,512),
        ResidualBlock(512,512,2)
    )
    self.avgpool = nn.AvgPool2d(2)
    # vowel_diacritic
    self.fc1 = nn.Linear(512,11)
    # grapheme_root
    self.fc2 = nn.Linear(512,168)
    # consonant_diacritic
    self.fc3 = nn.Linear(512,7)

def forward(self,x):
    x = self.block1(x)
    x = self.block2(x)
    x = self.block3(x)
    x = self.block4(x)
    x = self.block5(x)
    x = self.avgpool(x)
    x = x.view(x.size(0),-1)
    x1 = self.fc1(x)
    x2 = self.fc2(x)
    x3 = self.fc3(x)
    return x1,x2,x3

class ResidualBlock(nn.Module):

    def __init__(self,in_channels,out_channels,stride=1,kernel_size=3,padding=1,bias=False):
        super(ResidualBlock,self).__init__()
        self.cnn1 = nn.Sequential(nn.Conv2d(in_channels,out_channels,kernel_size,stride,padding,bias=False), nn.BatchNorm2d(out_channels), nn.ReLU(True) )

        self.cnn2 = nn.Sequential( nn.Conv2d(out_channels,out_channels,kernel_size,1,padding,bias=False), nn.BatchNorm2d(out_channels) )

        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential( nn.Conv2d(in_channels,out_channels,kernel_size=1,stride=stride,bias=False), nn.BatchNorm2d(out_channels) )
        else:
            self.shortcut = nn.Sequential()

    def forward(self,x):
        residual = x
        x = self.cnn1(x)
        x = self.cnn2(x)
        x += self.shortcut(residual)
        x = nn.ReLU(True)(x)
        return x

I am still new to PyTorch and I am trying to obtain the architectures for ResNets 34, 50 and 101 in a similar format as above for ResNet18. Initially I thought that the only changes that I have to make are the last parameter of each block according to the source GitHub code at https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py, where for ResNet34 we have return _resnet('resnet34', BasicBlock, [3, 4, 6, 3], pretrained, progress, **kwargs).

Am I doing this wrong? I am still reading up on the documentation and trying to make sense of the methodologies, so some guidance will be deeply appreciated. :')

Output:
Coming from a TensorFlow background with a flavour of FastAi: you can use all those  ResNet variants if you import them directly from torchvision module:

#Download the pretrained model
import torch
import torchvision.models as models
model = models.resnet18(pretrained = True) #change here to whatever model you want

#Switch device to gpu if available
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


Instruction: NameError: "linear regression" is not defined

Input:
Here is a code snippet where I am applying Linear regression using Pytorch. I face a NameError, that says name "linear regression" not defined. Kindly help in rectifying it.

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

x_data=Variable(torch.Tensor([[10.0],[9.0],[3.0],[2.0]]))
y_data=Variable(torch.Tensor([[90.0],[80.0],[50.0],[30.0]]))

class LinearRegression(torch.nn.Module):

  def __init__(self):
    super(LinearRegression,self). __init__ ()
    self.linear = torch.nn.Linear(1,1)

  def forward(self, x):
    y_pred = self.linear(x)
    return y_pred

  model = LinearRegression()

Output:
model = LinearRegression() should be outside class

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

x_data=Variable(torch.Tensor([[10.0],[9.0],[3.0],[2.0]]))
y_data=Variable(torch.Tensor([[90.0],[80.0],[50.0],[30.0]]))

class LinearRegression(torch.nn.Module):

  def __init__(self):
    super(LinearRegression,self). __init__ ()
    self.linear = torch.nn.Linear(1,1)

  def forward(self, x):
    y_pred = self.linear(x)
    return y_pred

model = LinearRegression()


Instruction: How to disable progress bar in Pytorch Lightning

Input:
I have a lot of issues withe the  tqdm progress bar in Pytorch Lightning:

when I run trainings in a terminal, the progress bars overwrite themselves. At the end of an training epoch, a validation progress bar is printed under the training bar, but when that ends, the progress bar from the next training epoch is printed over the one from the previous epoch. hence it's not possible to see the losses from previous epochs.

INFO:root:  Name    Type Params
0   l1  Linear    7 K
Epoch 2:  56%|████████████▊          | 2093/3750 [00:05<00:03, 525.47batch/s, batch_nb=1874, loss=0.714, training_loss=0.4, v_nb=51]

the progress bars wobbles from left to right, caused by the changing in number of digits behind the decimal point of some losses.
when running in Pycharm, the validation progress bar is not printed, but in stead,

INFO:root:  Name    Type Params
0   l1  Linear    7 K
Epoch 1:  50%|█████     | 1875/3750 [00:05<00:05, 322.34batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  50%|█████     | 1879/3750 [00:05<00:05, 319.41batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  52%|█████▏    | 1942/3750 [00:05<00:04, 374.05batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  53%|█████▎    | 2005/3750 [00:05<00:04, 425.01batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  55%|█████▌    | 2068/3750 [00:05<00:03, 470.56batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  57%|█████▋    | 2131/3750 [00:05<00:03, 507.69batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  59%|█████▊    | 2194/3750 [00:06<00:02, 538.19batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  60%|██████    | 2257/3750 [00:06<00:02, 561.20batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  62%|██████▏   | 2320/3750 [00:06<00:02, 579.22batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  64%|██████▎   | 2383/3750 [00:06<00:02, 591.58batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  65%|██████▌   | 2445/3750 [00:06<00:02, 599.77batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  67%|██████▋   | 2507/3750 [00:06<00:02, 605.00batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  69%|██████▊   | 2569/3750 [00:06<00:01, 607.04batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  70%|███████   | 2633/3750 [00:06<00:01, 613.98batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]

I would like to know if these issues can be solved or else how can I disable the progress bar and instead, just print some log details on the screen.

Output: Use the command show_progress_bar=False in Trainer.


Instruction: Python/Pytorch - how to use Image arrays?

Input:
I want to put image data in a neural network, but I am having trouble using the Image datatype.
I read my data here using Pytorch;

import torch
import torchvision
import numpy as np
from settings import Settings

class Data_Read:
    @staticmethod
    def getTrain():
        train_dataset = torchvision.datasets.ImageFolder(
            root=Settings.pathTrainImagesCopy,
            transform=torchvision.transforms.ToTensor()
        )
        train_loader = torch.utils.data.DataLoader(
            train_dataset,
            batch_size=64,
            num_workers=0,
            shuffle=True
        )
        return train_loader

    @staticmethod
    def getTest():
        test_dataset = torchvision.datasets.ImageFolder(
            root=Settings.pathTestImagesCopy,
            transform=torchvision.transforms.ToTensor()
        )
        test_loader = torch.utils.data.DataLoader(
            test_dataset,
            batch_size=64,
            num_workers=0,
            shuffle=True
        )
        return test_loader

The following code creates a single dimension column of images;

class Imagez:
    @staticmethod
    def Get(arr):
        imageData = []
        for item in arr:
            filePath = item
            img = Image.open(filePath).convert('LA')
            imageData.append(img)

        return imageData

And these methods are called from the main class as follows;

trainData = Data_Read.getTrain()
testData = Data_Read.getTest()

arrTrain = np.array(trainData.dataset.imgs)[:,0]
labelTrain = trainData.dataset.targets

arrTest = np.array(testData.dataset.imgs)[:,0]
labelTest = testData.dataset.targets

X_Train = Imagez.Get(arrTest)

I find whenever I try to use the Images datatype I in X_Train I get into trouble with error messages.
For example;

mlp = MLPClassifier(hidden_layer_sizes=(10,10,10), max_iter=1000 )
mlp.fit(X_Train, labelTrain)

Will give me this error message;

  Traceback (most recent call last):   File
  "c:\Users\hijik.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\ptvsd_launcher.py",
  line 43, in
      main(ptvsdArgs)   File "c:\Users\hijik.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\old_ptvsd\ptvsd__main__.py",
  line 432, in main
      run()   File "c:\Users\hijik.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\old_ptvsd\ptvsd__main__.py",
  line 316, in run_file
      runpy.run_path(target, run_name='main')   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\runpy.py", line
  263, in run_path
      pkg_name=pkg_name, script_name=fname)   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\runpy.py", line
  96, in _run_module_code
      mod_name, mod_spec, pkg_name, script_name)   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\runpy.py", line
  85, in _run_code
      exec(code, run_globals)   File "d:\702\702-Coursework-Task-5\src\Main.py", line 74, in
      mlp.fit(X_Train, Y_TrainLabels)   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py",
  line 981, in fit
      return self._fit(X, y, incremental=(self.warm_start and   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py",
  line 323, in _fit
      X, y = self._validate_input(X, y, incremental)   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py",
  line 919, in _validate_input
      multi_output=True)   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py",
  line 719, in check_X_y
      estimator=estimator)   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py",
  line 496, in check_array
      array = np.asarray(array, dtype=dtype, order=order)   File "C:\Users\hijik\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\core\numeric.py",
  line 538, in asarray
      return array(a, dtype, copy=False, order=order) TypeError: int() argument must be a string, a bytes-like object or a number, not
  'Image'

I am thinking I need to convert my images to another data type. What advice would you give?

EDIT - this is the minimum reproducible error;

X_Train = []
filePath = '..\\images\\Train\\anger\\S010_004_00000014.png'
img = Image.open(filePath).convert('LA')
X_Train.append(img)
Y_TrainLabels = ["0"]
mlp = MLPClassifier(hidden_layer_sizes=(10,10,10), max_iter=1000 )
mlp.fit(X_Train, Y_TrainLabels)

Output:
First of all, your Imagez class returns list of PIL images, those cannot be used for training as you need number representations.

Easy fix would be:

import torchvision

class Imagez:
    @staticmethod
    def Get(arr):
        imageData = []
        for item in arr:
            filePath = item
            # This transforms below
            img = torchvision.transforms.functional.to_tensor(
                Image.open(filePath).convert("LA")
            )
            imageData.append(img)

        return imageData

See convert docs and whether you really need it.

Secondly, if you are going for sklearn and it's MLPClassifier, you have transform those arrays into np.array.

For each you can call .numpy() on pytorch tensor to convert it and stack it afterwards into an array.

Furthermore, your inputs and outputs seem to be images, you either have to flatten those into a single vector for fully connected layers or use convolutional neural networks and PyTorch, for example the network below:

import torch

model = torch.nn.Sequential(
    torch.nn.Conv2d(3, 64, kernel_size=3),
    torch.nn.Conv2d(64, 128, kernel_size=3),
    torch.nn.Conv2d(128, 128, kernel_size=3),
    torch.nn.Conv2d(128, 128, kernel_size=3),
    torch.nn.Conv2d(128, 1, kernel_size=3),
)

You could find some basic tutorials on PyTorch website, 60 Minute Blitz seems like a good starting point for you cause there is a lot to fix here.


Instruction: Random Choice with Pytorch?

Input:
I have a tensor of pictures, and would like to randomly select from it. I'm looking for the equivalent of np.random.choice().

import torch

pictures = torch.randint(0, 256, (1000, 28, 28, 3))

Let's say I want 10 of these pictures.

Output:
torch has no equivalent implementation of np.random.choice(), see the discussion here. The alternative is indexing with a shuffled index or random integers.

To do it with replacement:

Generate n random indices
Index your original tensor with these indices

pictures[torch.randint(len(pictures), (10,))]

To do it without replacement:

Shuffle the index
Take the n first elements

indices = torch.randperm(len(pictures))[:10]

pictures[indices]

Read more about torch.randint and torch.randperm. Second code snippet is inspired by this post in PyTorch Forums.


Instruction: Finding indices with zeros from 4D Tensors in Pytorch

Input:
I have an interesting question for you, if you are working with Pytorch.

I have a batch of images with shape (3, 224, 224). So, if my batch size is 64, let’s say, the final tensor I have has the shape of (64, 3, 224, 224).

Now, here is the question. Suppose that some of the images in this batch is filled with only zeros. What is the fastest way to find out which batch indices are with only zeros?

I don’t want to create for loop for that, since it is slow.

Thanks for your answer.

Output:
a cheaper way to do this is assume only empty image would have sum = 0, which i think is pretty reasonable

import torch
t = torch.rand(64,3,224,224)
t[10] = 0
s = t.view(64, -1).sum(dim = -1)
zero_index = (s==0).nonzero().item() # 10