Instruction: Am I using polynomial regression right in pytorch?

Input:
I am trying to do polynomial regression by pytorch. First I just tried only linear regression (b + wx).

model_1 = RegressionModel()
W = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad = True)
optimizer_1 = torch.optim.SGD([W, b], lr = 0.001)

x_train = torch.FloatTensor(dataset.x_data['LSTAT'])
y_train = torch.FloatTensor(dataset.data['target'])

nb_epochs = 10000

for epoch in range(nb_epochs + 1):
    hypothesis = x_train * W + b
    cost = torch.nn.functional.mse_loss(hypothesis, y_train.float())

    optimizer_1.zero_grad()
    cost.backward()
    optimizer_1.step()

    print('Epoch {:4d}/{} W: {:.3f}, b: {:.3f}, Cost: {:.6f}'.format(epoch,
    nb_epochs, W.item(), b.item(), cost.item()))

Then I changed and added some variables to do polynomial regression (b + w1x + w2x^2)

model_2 = RegressionModel()
W1 = torch.zeros(1, requires_grad=True)
W2 = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad = True)
optimizer_2 = torch.optim.SGD([W2, W1, b], lr = 0.0000099)

x_train = torch.FloatTensor(dataset.x_data['LSTAT'])
y_train = torch.FloatTensor(dataset.data['target'])

nb_epochs = 10000

for epoch in range(nb_epochs + 1):
    hypothesis = b + x_train * W1 + x_train * x_train * W2
    cost = torch.nn.functional.mse_loss(hypothesis, y_train.float())

    optimizer_2.zero_grad()
    cost.backward()
    optimizer_2.step()

    print('Epoch {:4d}/{} W1: {:.3f}, W2: {:.3f}, b: {:.3f}, Cost:
    {:.6f}'.format(epoch, nb_epochs, W1.item(), W2.item(), b.item(),
    cost.item()))

Can I try polynomial regression like this? If not, I would be really appreciate if you let me know. I'm really noob to pytorch...

Output:
Your code should work. When working with larger data, it will be more efficient if you do the regression in a single matrix operation. For that, you need to first pre-compute polynomials of your input features:

x_train_polynomial = torch.stack([x_train, x_train ** 2], dim=1)

To save some lines, you can rewrite the projection a linear layer:

import torch.nn as nn
projection = nn.Linear(2, 1, bias=True)

In the training loop, you can call:

hypothesis = projection(x_train_polynomial)


Instruction: PyTorch loss function referencing model parameters

Input:
For an assignment, I need to create a movie recommender system that uses a supplied loss function:

sum(i=1,M) sum(j=1,M) indicator[i̸=j](viT vj − Xi,j )**2

What this means is, the dot product between two movie embeddings, Vi and Vj should be pretty darn close to Xi,j. Where Xi,j is the sum of users who liked both movie i and movie j. Entries where i == j are omitted by the indicator function, (set to 0.)

The assignment deliverable is a weight matrix from the hidden layer. Its dimensions should be 9724x300, where there are 9724 unique movie IDs, and 300 neurons. 300 is an arbitrary choice, influenced by use of 300 neurons in Google's word2vec.

What I have:

source_data: rows are users, columns are movies. A 1 in a given cell means that the corresponding user liked the corresponding movie (dislike = 0.)
preprocessed_data: matrix multiplication of source_data transposed on itself. (i.e. every cell is the sum of users who liked BOTH movie i and j. The diagonal entries are not of use to me, as they're just the sum of users who liked a single movie.)

Where I'm stuck:

Not sure how to define my own loss function in terms of i and j that can compare the sum of users who like movies i and j in preprocessed_data to rows, i and j, in hidden layer weight tensor.
Not sure how to organize my data into X and y tensors, such that the data is appropriate to the loss function.

Output:
Before you read further, please note that asking for and receiving direct help in your assignment from StackOverflow may be against the rules of your school and lead to consequences for you as a student!

That being said, the way I would model this problem is as follows:

import torch

U = 300 # number of users
M = 30  # number of movies
D = 4   # dimension of embedding vectors

source = torch.randint(0, 2, (U, M)) # users' ratings
X = source.transpose(0, 1) @ source  # your `preprocessed_data`

# initial values for your embedding. This is what your algorithm needs to learn
v = torch.randn(M, D, requires_grad=True)
X = X.to(torch.float32) # necessary to be in line with `v`

# this is the `(viT vj − Xi,j )**2` part
loss_elementwise = (v @ v.transpose(0, 1) - X).pow(2)

# now we need to get rid of the diagonal. Notice that we can equally
# well get rid of the diagonal and the whole upper triangular part,
# as well, since both V @ V.T and source.T @ source are symmetric, so
# the upper triangular part contains just
# a mirror reflection of the lower triangular part.
# This means that we actually implement a bit different summation:
# sum(i=1,M) sum(j=1,i-1) stuff(i, j)
# instead of
# sum(i=1,M) sum(j=1,M) indicator[i̸=j] stuff(i, j)
# and get exactly half the original value
masked = torch.tril(loss_elementwise, -1)

# finally we sum it up, multiplying by 2 to make up
# for the "lost" upper triangular part
loss = 2 * masked.sum()

Now what is left for you to implement is the optimization loop which will use gradient of loss to optimize the values of v.


Instruction: How to add L1 Regularization to PyTorch NN Model?

Input:
When searching for ways to implement L1 regularization in PyTorch Models, I came across this question, which is now 2 years old so i was wondering if theres anything new on this topic?
I also found this recent approach of dealing with the missing l1 function. However I don't understand how to use it for a basic NN as shown below.
class FFNNModel(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim, dropout_rate):
        super(FFNNModel, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        self.dropout_rate = dropout_rate
        self.drop_layer = nn.Dropout(p=self.dropout_rate)
        self.fully = nn.ModuleList()
        current_dim = input_dim
        for h_dim in hidden_dim:
            self.fully.append(nn.Linear(current_dim, h_dim))
            current_dim = h_dim
        self.fully.append(nn.Linear(current_dim, output_dim))

    def forward(self, x):
        for layer in self.fully[:-1]:
            x = self.drop_layer(F.relu(layer(x)))
        x = F.softmax(self.fully[-1](x), dim=0)
        return x

I was hoping simply putting this before training would work:
model = FFNNModel(30,5,[100,200,300,100],0.2)
regularizer = _Regularizer(model)
regularizer = L1Regularizer(regularizer, lambda_reg=0.1)

with
out = model(inputs)
loss = criterion(out, target) + regularizer.__add_l1()

Does anyone understand how to apply these 'ready to use' classes?

Output:
I haven't run the code in question, so please reach back if something doesn't exactly work. Generally, I would say that the code you linked is needlessly complicated (it may be because it tries to be generic and allow all the following kinds of regularization). The way it is to be used is, I suppose

model = FFNNModel(30,5,[100,200,300,100],0.2)
regularizer = L1Regularizer(model, lambda_reg=0.1)

and then

out = model(inputs)
loss = criterion(out, target) + regularizer.regularized_all_param(0.)

You can check that regularized_all_param will just iterate over parameters of your model and if their name ends with weight, it will accumulate their sum of absolute values. For some reason the buffer is to be manually initialized, that's why we pass in the 0..

Really though, if you wish to efficiently regularize L1 and don't need any bells and whistles, the more manual approach, akin to your first link, will be more readable. It would go like this

l1_regularization = 0.
for param in model.parameters():
    l1_regularization += param.abs().sum()
loss = criterion(out, target) + l1_regularization

This is really what is at heart of both approaches. You use the Module.parameters method to iterate over all model parameters and you sum up their L1 norms, which then becomes a term in your loss function. That's it. The repo you linked comes up with some fancy machinery to abstract it away but, judging by your question, fails :)


Instruction: How do you generate positive definite matrix in pytorch?

Input:
I am trying to define Multivariate Gaussian distribution with randomly generated covariance matrix:

psi = torch.zeros(512).normal_(0., 1.).requires_grad_(True)

# Generate random matrix
Sigma_k = torch.rand(512, 512)
# Make it symmetric positive
Sigma_k = Sigma_k * Sigma_k.t()
# Make it definite
Sigma_k.add_(0.001, torch.eye(512)).requires_grad_(True)

multivariate_normal.MultivariateNormal(psi, Sigma_k)

But I end up with getting an exception:

  RuntimeError: Lapack Error in potrf : the leading minor of order 2 is not positive definite at /Users/soumith/mc3build/conda-bld/pytorch_1549597882250/work/aten/src/TH/generic/THTensorLapack.cpp:658

What is the proper way to generate positive definite square matrix?

Output:
The answer is one should make a dot product of matrix A and it's transpose matrix (A.t()) in order to obtain a positive semi-definite matrix. The last thing is to ensure that it is definite (strictly greater than zero).

With Pytorch:

Sigma_k = torch.rand(512, 512)
Sigma_k = torch.mm(Sigma_k, Sigma_k.t())
Sigma_k.add_(torch.eye(512))

Formal algorithm is described here.


Instruction: How to use a different test batch size for RNN in PyTorch?

Input:
I want to train an RNN over 5 training points where each sequence also has a size of 5. At test time, I want to send in a single data point and compute the output.

The task is to predict the next character in a sequence of five characters (all encoded as 1-hot vectors). I have tried duplicating the test data point five times. However, I am sure that this is not the right way to solve this problem.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

# Define the parameters
H = [ 1, 0, 0, 0 ]
E = [ 0, 1, 0, 0 ]
L = [ 0, 0, 1, 0 ]
O = [ 0, 0, 0, 1 ]

# Define the model
net = nn.RNN(input_size=4, hidden_size=4, batch_first=True)

# Generate data
data = [[H,E,L,L,O],
        [E,L,L,O,H],
        [L,L,O,H,E],
        [L,O,H,E,L],
        [O,H,E,L,L]]
inputs = torch.tensor(data).float()
hidden = torch.randn(1,5,4) # Random initialization
correct_outputs = torch.tensor(np.array(data[1:]+[data[0]]).astype(float).tolist(), requires_grad=True)

# Set the loss function
criterion = torch.nn.MSELoss()

# Set the optimizer
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

# Perform gradient descent until convergence
for epoch in range(1000):
    # Forward Propagation
    outputs, hidden = net(inputs, hidden)
    # Compute and print loss
    loss = criterion(nn.functional.softmax(outputs,2), correct_outputs)
    print('epoch: ', epoch,' loss: ', loss.item())
    # Zero the gradients
    optimizer.zero_grad()
    # Backpropagation
    loss.backward(retain_graph=True)
    # Parameter update
    optimizer.step()

# Predict
net(torch.tensor([[H,E,L,L,O]]).float(),hidden)

I get the following error:

RuntimeError: Expected hidden size (1, 1, 4), got (1, 5, 4)

I understand that torch wants a tensor of size (1,1,4) but I am not sure how I can convert the initial hidden state from (1, 5, 4) to (1, 1, 4). Any help would be highly appreciated!

Output:
You are getting the error because you are using:

hidden = torch.randn(1,5,4) # Random initialization

Instead, you should use:

hidden = torch.randn(1,inputs.size(0),4) # Random initialization

to cope up with the batch size of the inputs. So, do the following:

# Predict
inputs = torch.tensor([[H,E,L,L,O]]).float()
hidden = torch.randn(1,inputs.size(0),4)
net(inputs, hidden)

Suggestion: improve your coding style by following some good examples in PyTorch.


Instruction: "Runtimeerror: bool value of tensor with more than one value is ambiguous" fastai

Input:
I am doing semantic segmentation using cityscapes dataset in fastai. I want to ignore some classes while calculating accuracy. this is how I defined accuracy according to fastai deep learning course:

name2id = {v:k for k,v in enumerate(classes)}
unlabeled = name2id['unlabeled']
ego_v = name2id['ego vehicle']
rectification = name2id['rectification border']
roi = name2id['out of roi']
static = name2id['static']
dynamic = name2id['dynamic']
ground = name2id['ground']

def acc_cityscapes(input, target):
    target = target.squeeze(1)

    mask=(target!=unlabeled and target!= ego_v and target!= rectification
    and target!=roi and target !=static and target!=dynamic and
    target!=ground)

return (input.argmax(dim=1)[mask]==target[mask]).float().mean()

This code works if I ignore only one of the classes:

mask=target!=unlabeled

But when I am trying to ignore multiple classes like this:

mask=(target!=unlabeled and target!= ego_v and target!= rectification
    and target!=roi and target !=static and target!=dynamic and
    target!=ground)

I am getting this error:

runtimeError: bool value of tensor with more than one value is ambiguous

Any Idea how do I solve this?

Output:
The problem is probably because your tensor contains more than 1 bool values, which will lead to an error when doing logical operation (and, or). For example,

>>> import torch
>>> a = torch.zeros(2)
>>> b = torch.ones(2)
>>> a == b
tensor([False, False])
>>> a == 0
tensor([True, True])
>>> a == 0 and True
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: bool value of Tensor with more than one value is ambiguous
>>> if a == b:
...     print (a)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: bool value of Tensor with more than one value is ambiguous

The potential solution might be using logical operator directly.

>>> (a != b) & (a == b)
tensor([False, False])

>>> mask = (a != b) & (a == b)
>>> c = torch.rand(2)
>>> c[mask]
tensor([])

Hope it helps.


Instruction: Extract sub tensor in PyTorch

Input:
For this tensor is PyTorch,

tensor([[ 0.7646,  0.5573,  0.4000,  0.2188,  0.7646,  0.5052,  0.2042,  0.0896,
          0.7667,  0.5938,  0.3167,  0.0917],
        [ 0.4271,  0.1354,  0.5000,  0.1292,  0.4260,  0.1354,  0.4646,  0.0917,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [ 0.7208,  0.5656,  0.3000,  0.1688,  0.7177,  0.5271,  0.1521,  0.0667,
          0.7198,  0.5948,  0.2438,  0.0729],
        [ 0.6292,  0.8250,  0.4000,  0.2292,  0.6271,  0.7698,  0.2083,  0.0812,
          0.6281,  0.8604,  0.3604,  0.0917]], device='cuda:0')

How can I extract to new Tensor for those values

0.7646,  0.5573,  0.4000,  0.2188
0.4271,  0.1354,  0.5000,  0.1292

How to get the first 4 of two rows into a new tensor?

Output:
Actually the question was answered from @zihaozhihao in the Comments but in case you are wondering where that comes from it would be helpful if you structured your Tensor like this:

x = torch.Tensor([
        [ 0.7646,  0.5573,  0.4000,  0.2188,  0.7646,  0.5052,  0.2042,  0.0896, 0.7667,  0.5938,  0.3167,  0.0917],
        [ 0.4271,  0.1354,  0.5000,  0.1292,  0.4260,  0.1354,  0.4646,  0.0917, -1.0000, -1.0000, -1.0000, -1.0000],
        [ 0.7208,  0.5656,  0.3000,  0.1688,  0.7177,  0.5271,  0.1521,  0.0667, 0.7198,  0.5948,  0.2438,  0.0729],
        [ 0.6292,  0.8250,  0.4000,  0.2292,  0.6271,  0.7698,  0.2083,  0.0812, 0.6281,  0.8604,  0.3604,  0.0917]

                ])

so now it is more clear that you have a shape (4, 12) you can think about it like an excel file, you have 4 rows and 12 columns. Now what you want is to extract from the two first rows the 4 first columns and that's why your solution would be:

x[:2, :4] # 2 means you want to take all the rows until the second row and then you set that you want all the columns until the fourth column, this Code will also give the same result x[0:2, 0:4]


Instruction: PyTorch matrix factorization embedding error

Input:
I'm trying to use a single hidden layer NN to perform matrix factorization. In general, I'm trying to solve for a tensor, V, with dimensions [9724x300] where there are 9724 items in inventory, and 300 is the arbitrary number of latent features.

The data that I have is a [9724x9724] matrix, X, where columns and rows represent the number of mutual likes. (eg X[0,1] represents the sum of users who like both item 0 and item 1. Diagonal entries are not of importance.

My goal is to use MSE loss, such that the dot product of V[i,:] on V[j,:] transposed is very, very close to X[i,j].

Below is code that I've adapted from the below link.

https://blog.fastforwardlabs.com/2018/04/10/pytorch-for-recommenders-101.html

import torch
from torch.autograd import Variable

class MatrixFactorization(torch.nn.Module):
    def __init__(self, n_items=len(movie_ids), n_factors=300):
        super().__init__()

        self.vectors = nn.Embedding(n_items, n_factors,sparse=True)

    def forward(self, i,j):
        return (self.vectors([i])*torch.transpose(self.vectors([j]))).sum(1)

    def predict(self, i, j):
        return self.forward(i, j)

model = MatrixFactorization(n_items=len(movie_ids),n_factors=300)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for i in range(len(movie_ids)):
    for j in range(len(movie_ids)):
    # get user, item and rating data
        rating = Variable(torch.FloatTensor([Xij[i, j]]))
        # predict
#         i = Variable(torch.LongTensor([int(i)]))
#         j = Variable(torch.LongTensor([int(j)]))
        prediction = model(i, j)
        loss = loss_fn(prediction, rating)

        # backpropagate
        loss.backward()

        # update weights
        optimizer.step()

The error returned is:

TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not list

I'm very new to embeddings. I had tried replacing embeddings as a simple float tensor, however the MatrixFactorization class, which I defined, did not recognize the tensor as a model parameters to be optimized over.

Any thoughts on where I'm going wrong?

Output:
You are passing a list to self.vectors,

return (self.vectors([i])*torch.transpose(self.vectors([j]))).sum(1)

Try to convert it to tensor before you call self.vectors()


Instruction: What is the time-complexity of the pseudo-inverse in pytorch (i.e. torch.pinverse)?

Input:
Let's say I have a matrix X with n, m == X.shape in PyTorch.
What is the time complexity of calculating the pseudo-inverse with torch.pinverse?

In other words, what is the time complexity of

X_p = torch.pinverse(X)

?

Here is the documentation

Output:
The PyTorch documentation states that pinverse is calculated using SVD (singular value decomposition). The complexity of SVD is O(n m^2), where m is the larger dimension of the matrix and n the smaller. Thus this is the complexity.

For more info, check out these pages on wikipedia:

https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse#Singular_value_decomposition_(SVD)
https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations#Matrix_algebra


Instruction: How to find built-in function source code in pytorch

Input:
I am trying to do research on batch normalization, and had to make some modifications for the pytorch BN code. I dig into the pytorch code and got stuck with torch.nn.functional.batch_norm, which references torch.batch_norm.

The problem is that torch.batch_norm cannot be further found in the torch library. Is there any way I can find the source code of this built-in function and re-implement it? Thanks!

Output:
It's there, but it's not defined in Python. They're defined in C++ in the aten/ directories.

For CPU, the implementation (one of them, it depends on whether or not the input is contiguous) is here: https://github.com/pytorch/pytorch/blob/420b37f3c67950ed93cd8aa7a12e673fcfc5567b/aten/src/ATen/native/Normalization.cpp#L61-L126

For CUDA, the implementation is here: https://github.com/pytorch/pytorch/blob/7aae51cdedcbf0df5a7a8bf50a947237ac4b3ee8/aten/src/ATen/native/cudnn/BatchNorm.cpp#L52-L143


Instruction: Pytorch CUDA error: invalid configuration argument

Input:
I recently added a new component to my loss function. Running the new code works on a CPU, but I get the following error when I run it on a GPU, clearly relating to the backward pass:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-56dcbddd5230> in <module>
     20 recall = Recall(N_RECALL_CAND, K)
     21 #run the model
---> 22 train_loss, val_loss = fit(triplet_train_loader, triplet_test_loader, model, loss_fn, optimizer, scheduler, N_EPOCHS, cuda, LOG_INT)
     23 #measure recall

~/thesis/trainer.py in fit(train_loader, val_loader, model, loss_fn, optimizer, scheduler, n_epochs, cuda, log_interval, metrics, start_epoch)
     24         scheduler.step()
     25         # Train stage
---> 26         train_loss, metrics, writer_train_index = train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics, writer, writer_train_index)
     27
     28         message = 'Epoch: {}/{}. Train set: Average loss: {:.4f}'.format(epoch + 1, n_epochs, train_loss)

~/thesis/trainer.py in train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics, writer, writer_train_index)
     80         losses.append(loss.item())
     81         total_loss += loss.item()
---> 82         loss.backward()
     83         optimizer.step()
     84

/opt/anaconda3/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    116                 products. Defaults to ``False``.
    117         """
--> 118         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    119
    120     def register_hook(self, hook):

/opt/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94
     95

RuntimeError: CUDA error: invalid configuration argument

Here is a copy of the code that causes it to break in the loss function:

def forward(self, anchor, positive, negative, model, size_average=True):
    #regular triplet loss. This works on GPU and CPU
    distance_positive = (anchor - positive).pow(2).sum(1)  # .pow(.5)
    distance_negative = (anchor - negative).pow(2).sum(1)  # .pow(.5)
    losses = F.relu(distance_positive - distance_negative + self.margin)

    #the additional component that causes the error. This will run on CPU but fails on GPU
    anchor_dists = torch.cdist(model.embedding_net.anchor_net.anchors, model.embedding_net.anchor_net.anchors)
    t = (self.beta * F.relu(self.rho - anchor_dists))
    regularization = t.sum() - torch.diag(t).sum()

    return regularization + losses.mean() if size_average else losses.sum()

The error occurs with a batch size of 1 and on the first backward pass. Answers from here suggest that it has to do with a lack of memory, but my model isn't particularly large:

TripletNet(
  (embedding_net): EmbeddingNet(
    (anchor_net): AnchorNet(anchors torch.Size([128, 192]), biases torch.Size([128]))
    (embedding): Sequential(
      (0): AnchorNet(anchors torch.Size([128, 192]), biases torch.Size([128]))
      (1): Tanh()
    )
  )
)

The available memory on my GPU is the 8GB, which is much smaller than the model and the size of the cdist result which is 128x128.

I have no idea how to begin debugging this. If it is the case that I'm running out of memory because its keeping track of intermediate states, how do I navigate around this? Any help is appreciated!

EDIT: Monitoring the GPU memory usage shows that I’m well under the memory limit when it crashes.

Output: As per this thread on the pytroch forums, upgrading to pytorch 1.5.0 should solve this issue


Instruction: Speech Recognition - how to split a sentence into words?

Input:
I'm new to Speech Recognition, and I'm looking for an approach to split a sentence (or multiple sentences) in the form of audio/wav files, into individual words? This sounds like a standard problem, so I'm wondering how people in the industry approach it.

ps: yes this question was asked three years ago, but I'm looking for an up-to-date answer using newer libraries (i.e. pytorch and tensorflow 2.0). Thanks!

Output:
This is not so trivial.

What you want is called an alignment. I.e. where each audio frame is aligned to a word (or subword, character, or better individual phonemes).

The most reasonable approach would need a standard conventional speech recognition system. The easiest would be to use a HMM system, either backed by old fashioned GMMs, or maybe by NNs (which is called hybrid HMM-NN model). This also requires a lexicon (mapping of phonemes to words). Usually you would use an existing implementation of all that, e.g. Kaldi or RASR, as this is not so simple to implement. I have not seen a pure TF implementation of that. This software then calculates the best possible alignment path through the HMM (i.e. which has the highest probability, according to the trained model). If you know the ground truth words, this is the Viterbi algorithm, to calculate this best path. Otherwise you would do some decoding (using beam search).

What you can also do, but this will be more hacky, and less good (for this task of getting an alignment): Use some of the end-to-end models, e.g. encoder-decoder with attention, or CTC. For encoder-decoder with attention, you can use the attention weights to get a good guess where the words are (and then you can maybe guess where the boundaries are). Similar for CTC. But this will not be accurate. But this is something you can implement easily in pure TF.

In any case, the implementation itself is not so much the hard part (although still not simple). You first need to understand all the theory behind that. And maybe StackOverflow is not the right place to ask about that. Read through the Kaldi or RASR documentation maybe, or watch some lecture about speech recognition, or read a book about that topic.


Instruction: Why is my Fully Convolutional Autoencoder not symmetric?

Input:
I am developing a Fully Convolutional Autoencoder which takes 3 channels as input and outputs 2 channels (in: LAB, out: AB). Because the output should be the same size as the input, I use Full Convolution.

The Code:

import torch.nn as nn

class AE(nn.Module):
   def __init__(self):
       super(AE, self).__init__()

        self.encoder = nn.Sequential(
           # conv 1
           nn.Conv2d(in_channels=3, out_channels=64, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(64),
           nn.ReLU(),
           nn.MaxPool2d(kernel_size=2, stride=2),

           # conv 2
           nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(128),
           nn.ReLU(),
           nn.MaxPool2d(kernel_size=2, stride=2),

           # conv 3
           nn.Conv2d(in_channels=128, out_channels=256, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(256),
           nn.ReLU(),
           nn.MaxPool2d(kernel_size=2, stride=2),

           # conv 4
           nn.Conv2d(in_channels=256, out_channels=512, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(512),
           nn.ReLU(),
           nn.MaxPool2d(kernel_size=2, stride=2),

           # conv 5
           nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(1024),
           nn.ReLU()

       )

       self.decoder = nn.Sequential(
           # conv 6
           nn.ConvTranspose2d(in_channels=1024, out_channels=512, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(512),
           nn.ReLU(),

           # conv 7
           nn.Upsample(scale_factor=2, mode='bilinear'),
           nn.ConvTranspose2d(in_channels=512, out_channels=256, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(256),
           nn.ReLU(),

           # conv 8
           nn.Upsample(scale_factor=2, mode='bilinear'),
           nn.ConvTranspose2d(in_channels=256, out_channels=128, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(128),
           nn.ReLU(),

           # conv 9
           nn.Upsample(scale_factor=2, mode='bilinear'),
           nn.ConvTranspose2d(in_channels=128, out_channels=64, kernel_size=5, stride=1, padding=1),
           nn.BatchNorm2d(64),
           nn.ReLU(),

           # conv 10 out
           nn.Upsample(scale_factor=2, mode='bilinear'),
           nn.ConvTranspose2d(in_channels=64, out_channels=2, kernel_size=5, stride=1, padding=1),
           nn.Softmax()    # multi-class classification

           # TODO softmax deprecated
       )

   def forward(self, x):
       x = self.encoder(x)
       x = self.decoder(x)
       return x

The size the output tensor should be: torch.Size([1, 2, 199, 253])

The size the output tensor really has: torch.Size([1, 2, 190, 238])

My main problem is combining Conv2d and MaxPool2d and to set the correct parameter values in the ConvTranspose2d. Because of that, I treat those separately using the Upsample function for the MaxPool2d and ConvTranspose2d only for Conv2d. But I still have a little asymmetry and I really don't know why.

Thank you for the help!

Output:
There are two issues.

First is insufficient padding: with kernel_size=5 your convolutions are shrinking the image by 4 every time they are applied (2 pixels on each side), so you need padding=2, and not just 1, in all places.

Second is the "uneven" input size. What I mean is that once your convolutions are properly padded, you are left with downsampling operations which at each point try to divide your image resolution in half. When they fail, they just return a smaller result (integer division discards the remainder). Since your network has 4 successive 2x downsampling operations, you need your input to have H, W dimensions which are multiples of 2^4=16. Then you will actually get equally shaped output. An example below

import torch
import torch.nn as nn

class AE(nn.Module):
    def __init__(self):
        super(AE, self).__init__()

        self.encoder = nn.Sequential(
            # conv 1
            nn.Conv2d(in_channels=3, out_channels=64, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),

            # conv 2
            nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),

            # conv 3
            nn.Conv2d(in_channels=128, out_channels=256, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),

            # conv 4
            nn.Conv2d(in_channels=256, out_channels=512, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),

            # conv 5
            nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(1024),
            nn.ReLU()
        )

        self.decoder = nn.Sequential(
            # conv 6
            nn.ConvTranspose2d(in_channels=1024, out_channels=512, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(512),
            nn.ReLU(),

            # conv 7
            nn.Upsample(scale_factor=2, mode='bilinear'),
            nn.ConvTranspose2d(in_channels=512, out_channels=256, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(256),
            nn.ReLU(),

            # conv 8
            nn.Upsample(scale_factor=2, mode='bilinear'),
            nn.ConvTranspose2d(in_channels=256, out_channels=128, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(128),
            nn.ReLU(),

            # conv 9
            nn.Upsample(scale_factor=2, mode='bilinear'),
            nn.ConvTranspose2d(in_channels=128, out_channels=64, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(64),
            nn.ReLU(),

            # conv 10 out
            nn.Upsample(scale_factor=2, mode='bilinear'),
            nn.ConvTranspose2d(in_channels=64, out_channels=2, kernel_size=5, stride=1, padding=2),
            nn.Softmax()    # multi-class classification
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

input = torch.randn(1, 3, 6*16, 7*16)
output = AE()(input)
print(input.shape)
print(output.shape)


Instruction: Slightly adapt L1 loss to a weighted L1 loss in Pytorch, does gradient computation still work properly?

Input:
I implemented a neural network in Pytorch and I would like to use a weighted L1 loss function to train the network.

The implementation with the regular L1 loss contains this code for each epoch:

optimiser.zero_grad()
net.train()
_,forecast = net(torch.tensor(feature, dtype=torch.float).to(DEVICE))
loss = F.l1_loss(forecast, torch.tensor(target,dtype=torch.float).to(DEVICE),reduction='mean')
loss.backward()
params.append(net.parameters())
optimiser.step()

Now I want to use a weighted L1 loss instead. So I thought to use the same standard Pytorch L1 function again and rescale the forecasts and targets with weights. Will the gradient computation still be done correctly?

optimiser.zero_grad()
net.train()
_,forecast = net(torch.tensor(feature, dtype=torch.float).to(DEVICE))
loss = F.l1_loss(torch.t(torch.mul(torch.t(forecast),
                                           torch.tensor(weight,dtype=torch.float).to(DEVICE))) ,
                         torch.t(torch.mul(torch.t(torch.tensor(target,dtype=torch.float).to(DEVICE)),
                                           torch.tensor(weight,dtype=torch.float).to(DEVICE))),reduction='mean')
loss.backward()
params.append(net.parameters())
optimiser.step()

Output:
Yes, it will be correct.

If you are not using in-place operations, the gradients will be computed correctly. Besides, in the current version of Pytorch, there will be an error reported if you accidentally involve some in-place operations into your program.

Here is a related discussion. You can find more information here.


Instruction: Multilabel classification with class imbalance in Pytorch

Input:
I have a multilabel classification problem, which I am trying to solve with CNNs in Pytorch. I have 80,000 training examples and 7900 classes; every example can belong to multiple classes at the same time, mean number of classes per example is 130.

The problem is that my dataset is very imbalance. For some classes, I have only ~900 examples, which is around 1%. For “overrepresented” classes I have ~12000 examples (15%). When I train the model I use BCEWithLogitsLoss from pytorch with a positive weights parameter. I calculate the weights the same way as described in the documentation: the number of negative examples divided by the number of positives.

As a result, my model overestimates almost every class… Mor minor and major classes I get almost twice as many predictions as true labels. And my AUPRC is just 0.18. Even though it’s much better than no weighting at all, since in this case the model predicts everything as zero.

So my question is, how do I improve the performance? Is there anything else I can do? I tried different batch sampling techniques (to oversample minority class), but they don’t seem to work.

Output:
I would suggest either one of these strategies
Focal Loss
A very interesting approach for dealing with un-balanced training data through tweaking of the loss function was introduced in
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He and Piotr Dollar Focal Loss for Dense Object Detection (ICCV 2017).
They propose to modify the binary cross entropy loss in a way that decrease the loss and gradient of easily classified examples while "focusing the effort" on examples where the model makes gross errors.
Hard Negative Mining
Another popular approach is to do "hard negative mining"; that is, propagate gradients only for part of the training examples - the "hard" ones.
see, e.g.:
Abhinav Shrivastava, Abhinav Gupta and Ross Girshick Training Region-based Object Detectors with Online Hard Example Mining (CVPR 2016)


Instruction: RuntimeError: expected device cpu and dtype Byte but got device cpu and dtype Bool

Input:
As described in the issue I opened, I get the following error when running the Pytorch inverse-cooking model on CPU:

RuntimeError: expected device cpu and dtype Byte but got device cpu and dtype Bool

I have tried running the demo.ipynb file in both my laptop's Intel i7-4700HQ 8 threads and my desktop Ryzen 3700x. I was using Arch Linux on my laptop and Manjaro on my desktop.

The model works fine when I run it on Google Collabs GPU.

According to the demo.ipynb file the model should be able to run on CPU as well. Does anyone know if I have to tweak any parameters in order to make it work?

Output:
As stated by @iacolippo and in the comment session and myDennisCode, the problem really was dependency versions. I had torchvision==0.4.0 (which confused me) and torch==1.2.0.

To fix the problem, simply install torch==0.4.1 and torchvision==0.2.1.


Instruction: How to use torchvision.transforms for data augmentation of segmentation task in Pytorch?

Input:
I am a little bit confused about the data augmentation performed in PyTorch.

Because we are dealing with segmentation tasks, we need data and mask for the same data augmentation, but some of them are random, such as random rotation.

Keras provides a random seed guarantee that data and mask do the same operation, as shown in the following code:

    data_gen_args = dict(featurewise_center=True,
                         featurewise_std_normalization=True,
                         rotation_range=25,
                         horizontal_flip=True,
                         vertical_flip=True)

    image_datagen = ImageDataGenerator(**data_gen_args)
    mask_datagen = ImageDataGenerator(**data_gen_args)

    seed = 1
    image_generator = image_datagen.flow(train_data, seed=seed, batch_size=1)
    mask_generator = mask_datagen.flow(train_label, seed=seed, batch_size=1)

    train_generator = zip(image_generator, mask_generator)

I didn't find a similar description in the official Pytorch documentation, so I don't know how to ensure that data and mask can be processed synchronously.

Pytorch does provide such a function, but I want to apply it to a custom Dataloader.

For example：

def __getitem__(self, index):
    img = np.zeros((self.im_ht, self.im_wd, channel_size))
    mask = np.zeros((self.im_ht, self.im_wd, channel_size))

    temp_img = np.load(Image_path + '{:0>4}'.format(self.patient_index[index]) + '.npy')
    temp_label = np.load(Label_path + '{:0>4}'.format(self.patient_index[index]) + '.npy')

    for i in range(channel_size):
        img[:,:,i] = temp_img[self.count[index] + i]
        mask[:,:,i] = temp_label[self.count[index] + i]

    if self.transforms:
        img = np.uint8(img)
        mask = np.uint8(mask)
        img = self.transforms(img)
        mask = self.transforms(mask)

    return img, mask

In this case, img and mask will be transformed separately, because some operations such as random rotation are random, so the correspondence between mask and image may be changed. In other words, the image may have rotated but the mask did not do this.

EDIT 1

I used the method in augmentations.py, but I got an error:：

Traceback (most recent call last):
  File "test_transform.py", line 87, in <module>
    for batch_idx, image, mask in enumerate(train_loader):
  File "/home/dirk/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 314, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/dirk/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 314, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/dirk/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 103, in __getitem__
    return self.dataset[self.indices[idx]]
  File "/home/dirk/home/data/dirk/segmentation_unet_pytorch/data.py", line 164, in __getitem__
    img, mask = self.transforms(img, mask)
  File "/home/dirk/home/data/dirk/segmentation_unet_pytorch/augmentations.py", line 17, in __call__
    img, mask = a(img, mask)
TypeError: __call__() takes 2 positional arguments but 3 were given

This is my code for __getitem__()：

data_transforms = {
    'train': Compose([
        RandomHorizontallyFlip(),
        RandomRotate(degree=25),
        transforms.ToTensor()
    ]),
}

train_set = DatasetUnetForTestTransform(fold=args.fold, random_index=args.random_index,transforms=data_transforms['train'])

# __getitem__ in class DatasetUnetForTestTransform
def __getitem__(self, index):
    img = np.zeros((self.im_ht, self.im_wd, channel_size))
    mask = np.zeros((self.im_ht, self.im_wd, channel_size))
    temp_img = np.load(Label_path + '{:0>4}'.format(self.patient_index[index]) + '.npy')
    temp_label = np.load(Label_path + '{:0>4}'.format(self.patient_index[index]) + '.npy')
    temp_img, temp_label = crop_data_label_from_0(temp_img, temp_label)
    for i in range(channel_size):
        img[:,:,i] = temp_img[self.count[index] + i]
        mask[:,:,i] = temp_label[self.count[index] + i]

    if self.transforms:
        img = T.ToPILImage()(np.uint8(img))
        mask = T.ToPILImage()(np.uint8(mask))
        img, mask = self.transforms(img, mask)

    img = T.ToTensor()(img).copy()
    mask = T.ToTensor()(mask).copy()
    return img, mask

EDIT 2

I found that after ToTensor, the dice between the same labels becomes 255 instead of 1, how to fix it?

# Dice computation
def DSC_computation(label, pred):
    pred_sum = pred.sum()
    label_sum = label.sum()
    inter_sum = np.logical_and(pred, label).sum()
    return 2 * float(inter_sum) / (pred_sum + label_sum)

Feel free to ask if more code is needed to explain the problem.

Output:
torchvision also provides similar functions [document].

Here is a simple example,

import torchvision
from torchvision import transforms

trans = transforms.Compose([transforms.CenterCrop((178, 178)),
                                    transforms.Resize(128),
                                    transforms.RandomRotation(20),
                                    transforms.ToTensor(),
                                    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
dset = torchvision.datasets.MNIST(data_root, transforms=trans)

EDIT

A brief example when customizing your own CelebA dataset. Note that, to apply transformations, you need call transform list in __getitem__.

class CelebADataset(Dataset):
    def __init__(self, root, transforms=None, num=None):
        super(CelebADataset, self).__init__()

        self.img_root = os.path.join(root, 'img_align_celeba')
        self.attr_root = os.path.join(root, 'Anno/list_attr_celeba.txt')
        self.transforms = transforms

        df = pd.read_csv(self.attr_root, sep='\s+', header=1, index_col=0)
        #print(df.columns.tolist())
        if num is None:
            self.labels = df.values
            self.img_name = df.index.values
        else:
            self.labels = df.values[:num]
            self.img_name = df.index.values[:num]

    def __getitem__(self, index):
        img = Image.open(os.path.join(self.img_root, self.img_name[index]))
        # only use blond_hair, eyeglass, male, smile
        indices = [9, 15, 20, 31]
        label = np.take(self.labels[index], indices)
        label[label==-1] = 0

        if self.transforms is not None:
            img = self.transforms(img)

        return np.asarray(img), label

    def __len__(self):
        return len(self.labels)

EDIT 2

I probably miss something at the first glance. The main point of your problem is how to apply "the same" data preprocessing to img and labels. To my understanding, there is no available Pytorch built-in function. So, what I did before is to implement the augmentation by myself.

class RandomRotate(object):
    def __init__(self, degree):
        self.degree = degree

    def __call__(self, img, mask):
        rotate_degree = random.random() * 2 * self.degree - self.degree
        return img.rotate(rotate_degree, Image.BILINEAR),
                           mask.rotate(rotate_degree, Image.NEAREST)

Note that the input should be PIL format. See this for more information.


Instruction: Get total amount of free GPU memory and available using pytorch

Input: I'm using google colab free Gpu's for experimentation and wanted to know how much GPU Memory available to play around, torch.cuda.memory_allocated() returns the current GPU memory occupied, but how do we determine total available memory using PyTorch.

Output:
In the recent version of PyTorch you can also use torch.cuda.mem_get_info:
https://pytorch.org/docs/stable/generated/torch.cuda.mem_get_info.html#torch.cuda.mem_get_info


Instruction: How do I effectively parallelize AlphaZero on the GPU?

Input:
I'm implementing a version of AlphaZero (AlphaGo's most recent incarnation) to be applied to some other domain.

The crux of the algorithm is a Monte Carlo Tree Search of the state space (CPU) interleaved with 'intuition' (probabilities) from a neural network in eval mode (GPU). The MCTS result is then used to train the neural network.

I already parallelized the CPU execution by launching multiple processes which each build up their own tree. This is effective and has now lead to a GPU bottleneck! (nvidia-smi showing the GPU at 100% all the time)

I have devised 2 strategies to parallelize GPU evaluations, however both of them have problems.

Each process evaluates the network only on batches from its own tree. In my initial naive implementation, this meant a batch size of 1. However, by refactoring some code and adding a 'virtual loss' to discourage (but not completely block) the same node from being picked twice we can get larger batches of size 1-4. The problem here is that we cannot allow large delays until we evaluate the batch or accuracy suffers, so a small batch size is key here.
Send the batches to a central "neural network worker" thread which combines and evaluates them. This could be done in a large batch of 32 or more, so the GPU could be used very efficiently. The problem here is that the tree workers send CUDA tensors 'round-trip' which is not supported by PyTorch. It is supported if I clone them first, but all that constant copying makes this approach slower than the first one.

I was thinking maybe a clever batching scheme that I'm not seeing could make the first approach work. Using multiple GPUs could speed up the first approach too, but the kind of parallelism I want is not natively supported by PyTorch. Maybe keeping all tensors in the NN worker and only sending ids around could improve the second approach, however the difficulty here is how to synchronize effectively to get a large batch without making the CPU threads wait too long.

I found next to no information on how AlphaZero or AlphaGo Zero were parallelized in their respective papers. I was able to find limited information online however which lead me to improve the first approach.

I would be grateful for any advice on this, particularly if there's some point or approach I missed.

Output:
Use tensorflow serving as an example, The prediction service could run in a different process, running a service to receive request from the worker (runs a MCTS process and send prediction request to the prediction service). We can keep a dict from the socket address to the socket itself.

The prediction service could read each query body and their header(which is different for each query), we can put that headers in a queue. While wait for maybe at most 100ms or the current batch is bigger than a batch size, The prediction runs. After the GPU gives the results, we loop over the results and as the order is the same as the headers in the queue, we can send back the responses via the socket based on each header (can be looked up from the dict we kept above).

As each query comes with a different header, you can not miss the request, response and the socket. While you can run a tensorflow serving with a GPU card while running multiple worker to keep the batch size big enough to get a larger throughput.

I aslo find a batching mechanism here in this repo:
https://github.com/richemslie/galvanise_zero


Instruction: How to implement contractive autoencoder in Pytorch?

Input:
I'm trying to create a contractive autoencoder in Pytorch. I found this thread and tried according to that. This is the snippet I wrote based on the mentioned thread:
import datetime
import numpy as np
import torch
import torchvision
from torchvision import datasets, transforms
from torchvision.utils import save_image, make_grid
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import matplotlib.pyplot as plt
%matplotlib inline

dataset_train = datasets.MNIST(root='MNIST',
                               train=True,
                               transform = transforms.ToTensor(),
                               download=True)
dataset_test  = datasets.MNIST(root='MNIST',
                               train=False,
                               transform = transforms.ToTensor(),
                               download=True)
batch_size = 128
num_workers = 2
dataloader_train = torch.utils.data.DataLoader(dataset_train,
                                               batch_size = batch_size,
                                               shuffle=True,
                                               num_workers = num_workers,
                                               pin_memory=True)

dataloader_test = torch.utils.data.DataLoader(dataset_test,
                                               batch_size = batch_size,
                                               num_workers = num_workers,
                                               pin_memory=True)

def view_images(imgs, labels, rows = 4, cols =11):
    imgs = imgs.detach().cpu().numpy().transpose(0,2,3,1)
    fig = plt.figure(figsize=(8,4))
    for i in range(imgs.shape[0]):
        ax = fig.add_subplot(rows, cols, i+1, xticks=[], yticks=[])
        ax.imshow(imgs[i].squeeze(), cmap='Greys_r')
        ax.set_title(labels[i].item())

# now let's view some
imgs, labels = next(iter(dataloader_train))
view_images(imgs, labels,13,10)

class Contractive_AutoEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Linear(784, 512)
        self.decoder = nn.Linear(512, 784)

    def forward(self, input):
        # flatten the input
        shape = input.shape
        input = input.view(input.size(0), -1)
        output_e = F.relu(self.encoder(input))
        output = F.sigmoid(self.decoder(output_e))
        output = output.view(*shape)
        return output_e, output

def loss_function(output_e, outputs, imgs, device):
    output_e.backward(torch.ones(output_e.size()).to(device), retain_graph=True)
    criterion = nn.MSELoss()
    assert outputs.shape == imgs.shape ,f'outputs.shape : {outputs.shape} != imgs.shape : {imgs.shape}'

    imgs.grad.requires_grad = True
    loss1 = criterion(outputs, imgs)
    print(imgs.grad)
    loss2 = torch.mean(pow(imgs.grad,2))
    loss = loss1 + loss2
    return loss

epochs = 50
interval = 2000
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Contractive_AutoEncoder().to(device)
optimizer = optim.Adam(model.parameters(), lr =0.001)

for e in range(epochs):
    for i, (imgs, labels) in enumerate(dataloader_train):
        imgs = imgs.to(device)
        labels = labels.to(device)

        outputs_e, outputs = model(imgs)
        loss = loss_function(outputs_e, outputs, imgs,device)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        if i%interval:
            print('')

    print(f'epoch/epoechs: {e}/{epochs} loss : {loss.item():.4f} ')

For the sake of brevity I just used one layer for the encoder and the decoder. It should work regardless of number of layers in either of them obviously!
But the catch here is, aside from the fact that I don't know if this is the correct way of doing this, (calculating gradients with respect to the input), I get an error which makes the former solution wrong/not applicable.
That is:
    imgs.grad.requires_grad = True
produces the error :

AttributeError : 'NoneType' object has no attribute 'requires_grad'

I also tried the second method suggested in that thread which is as follows:
class Contractive_Encoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Linear(784, 512)

    def forward(self, input):
        # flatten the input
        input = input.view(input.size(0), -1)
        output_e = F.relu(self.encoder(input))
        return output_e

class Contractive_Decoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.decoder = nn.Linear(512, 784)

    def forward(self, input):
        # flatten the input
        output = F.sigmoid(self.decoder(input))
        output = output.view(-1,1,28,28)
        return output

epochs = 50
interval = 2000
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model_enc = Contractive_Encoder().to(device)
model_dec = Contractive_Decoder().to(device)

optimizer = optim.Adam([{"params":model_enc.parameters()},
                        {"params":model_dec.parameters()}], lr =0.001)

optimizer_cond = optim.Adam(model_enc.parameters(), lr = 0.001)

criterion = nn.MSELoss()

for e in range(epochs):
    for i, (imgs, labels) in enumerate(dataloader_train):
        imgs = imgs.to(device)
        labels = labels.to(device)

        outputs_e = model_enc(imgs)
        outputs = model_dec(outputs_e)
        loss_rec = criterion(outputs, imgs)
        optimizer.zero_grad()
        loss_rec.backward()
        optimizer.step()

        imgs.requires_grad_(True)
        y = model_enc(imgs)
        optimizer_cond.zero_grad()
        y.backward(torch.ones(imgs.view(-1,28*28).size()))

        imgs.grad.requires_grad = True
        loss = torch.mean([pow(imgs.grad,2)])
        optimizer_cond.zero_grad()
        loss.backward()
        optimizer_cond.step()

        if i%interval:
            print('')

    print(f'epoch/epoechs: {e}/{epochs} loss : {loss.item():.4f} ')

but I face the error :
RuntimeError: invalid gradient at index 0 - got [128, 784] but expected shape compatible with [128, 512]

How should I go about this in Pytorch?

Output:
Summary

The final implementation for contractive loss that I wrote is as follows:

def loss_function(output_e, outputs, imgs, lamda = 1e-4, device=torch.device('cuda')):

    criterion = nn.MSELoss()
    assert outputs.shape == imgs.shape ,f'outputs.shape : {outputs.shape} != imgs.shape : {imgs.shape}'
    loss1 = criterion(outputs, imgs)

    output_e.backward(torch.ones(outputs_e.size()).to(device), retain_graph=True)
    # Frobenious norm, the square root of sum of all elements (square value)
    # in a jacobian matrix
    loss2 = torch.sqrt(torch.sum(torch.pow(imgs.grad,2)))
    imgs.grad.data.zero_()
    loss = loss1 + (lamda*loss2)
    return loss

and inside training loop you need to do:

for e in range(epochs):
    for i, (imgs, labels) in enumerate(dataloader_train):
        imgs = imgs.to(device)
        labels = labels.to(device)

        imgs.retain_grad()
        imgs.requires_grad_(True)

        outputs_e, outputs = model(imgs)
        loss = loss_function(outputs_e, outputs, imgs, lam,device)

        imgs.requires_grad_(False)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f'epoch/epochs: {e}/{epochs} loss: {loss.item():.4f}')

Full explanation

As it turns out and rightfully @akshayk07 pointed out in the comments, the implementation found in Pytorch forum was wrong in multiple places. The notable thing, being it wasn't implementing the actual contractive loss that was introduced in Contractive Auto-Encoders:Explicit Invariance During Feature Extraction paper! and also aside from that, the implementation wouldn't work at all for obvious reasons that will be explained in a moment.

The changes are obvious so I try to explain what's going on here. First of all note that imgs is not a leaf node, so the gradients would not be retained in the image .grad attribute.

In order to retain gradients for non leaf nodes, you should use retain_graph(). grad is only populated for leaf Tensors. Also imgs.retain_grad() should be called before doing forward() as it will       instruct the autograd to store grads into non-leaf nodes.

Update

Thanks to @Michael for pointing out that the correct calculation of Frobenius Norm is actually (from ScienceDirect):

  the square root of the sum of the squares of all the matrix entries

and not

  the the square root of the sum of the absolute values of all the
  matrix entries as explained here


Instruction: Size Mismatch using pytorch when trying to train data

Input:
I am really new to pytorch and just trying to use my own dataset to do a simple Linear Regression Model. I am only using the numbers values as inputs, too.

I have imported the data from the CSV

dataset = pd.read_csv('mlb_games_overview.csv')

I have split the data into four parts X_train, X_test, y_train, y_test

X = dataset.drop(['date', 'team', 'runs', 'win'], 1)
y = dataset['win']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=True)

I have converted the data to pytorch tensors

X_train = torch.from_numpy(np.array(X_train))
X_test = torch.from_numpy(np.array(X_test))
y_train = torch.from_numpy(np.array(y_train))
y_test = torch.from_numpy(np.array(y_test))

I have created a LinearRegressionModel

class LinearRegressionModel(torch.nn.Module):
    def __init__(self):
        super(LinearRegressionModel, self).__init__()
        self.linear = torch.nn.Linear(1, 1)
    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

I have initialized the optimizer and the loss function

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Now when I start to train the data I get the runtime error mismatch

EPOCHS = 500
for epoch in range(EPOCHS):
    pred_y = model(X_train) # RUNTIME ERROR HERE
    loss = criterion(pred_y, y_train)
    optimizer.zero_grad() # zero out gradients to update parameters correctly
    loss.backward() # backpropagation
    optimizer.step() # update weights
    print('epoch {}, loss {}'. format(epoch, loss.data[0]))

Error Log:

RuntimeError                              Traceback (most recent call last)
<ipython-input-40-c0474231d515> in <module>
  1 EPOCHS = 500
  2 for epoch in range(EPOCHS):
----> 3     pred_y = model(X_train)
  4     loss = criterion(pred_y, y_train)
  5     optimizer.zero_grad() # zero out gradients to update parameters correctly
RuntimeError: size mismatch, m1: [3540 x 8], m2: [1 x 1] at
C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:752

Output:
In your Linear Regression model, you have:

self.linear = torch.nn.Linear(1, 1)

But your training data (X_train) shape is 3540 x 8 which means you have 8 features representing each input example. So, you should define the linear layer as follows.

self.linear = torch.nn.Linear(8, 1)

A linear layer in PyTorch has parameters, W and b. If you set the in_features to 8 and out_features to 1, then the shape of the W matrix will be 1 x 8 and the length of b vector will be 1.

Since your training data shape is 3540 x 8, you can perform the following operation.

linear_out = X_train W_T + b

I hope it clarifies your confusion.


Instruction: Pytorch variable changes numpy variable even though memory addresses are not same

Input:
I have the following bit of code:

a = torch.ones(10); b = a.numpy()
a[0] += 1
print(a, b)

Both variables essentially hold the same values even though I only modified a. However, I checked the memory addresses of a and b using hex(id(a)) and they're different. So in this case, is b a pointer to a? What's going on?

Output:
Actually the raw data is at the same address. You can check like this,

a.storage().data_ptr()

Out[16]: 93866530123392

b.__array_interface__['data']

Out[17]: (93866530123392, False)


Instruction: Create self attributes in python using for loop when using pytorch

Input:
Within pytorch, creating layers, can be semi automated, thus the reason for using a for loop.

One of the main issues is that these layers cannot stored within a list or dictionary or else back propagation will not work.

Thus the reason for a work around.

Within the object, assigning new self attributes

How do i replace this

self.res1 = 1
self.res2 = 2
self.res3 = 3

with this

for i in range(2):
  res_name = 'res'+str(i+1)
  self.res_name = i

Now that i have created objects this way, how can I access them in the same way. For example, if we assume self.res_name is now an object?

for i in range(2):
   res_name = 'res'+str(i+1)
   out = self.res_name(out)

Output:
You probably should use a dict or list instead. But if you really want this for some reason, you can try setattr(x, attr, 'magic').

Thus, in your case, it's

for i in range(1, 4):
    res_name = 'res' + str(i)
    setattr(self, res_name, i)

See this related question for more info.


Instruction: Efficiency of .to(device) in pytorch

Input:
For achieving maximum efficiency, which command would be more efficient?

x = torch.randn(100, 100).to(device)
x = torch.randn(100, 100, device = device)

Is there a benefit to using one versus the other when doing heavy tensor operations? I was told one of them is less efficient, but can't properly figure how to compare the two.

I'm assuming the second is better as it directly exports to device instead of first creating a tensor and then having to transfer it to device.

Output:
well, I'm no expert in this. Here's the timing result for you.

%timeit torch.randn(100, 100).to(device)

The slowest run took 12.65 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 129 µs per loop

%timeit torch.randn(100, 100, device = device)

The slowest run took 88.54 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 11.6 µs per loop

P.S. I executed both these commands on Google Colab.


Instruction: Error in ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

Input:
The input image size is 512*512,in order to suit to the input of resnet.input image
I used

_img = Image.open(self.images[index]).convert('RGB')

in dataloader.
I used resnet50 as my network's backbone without fc.The output shape is

[4,2048,16,16]

then used two (conv bn relu) and a interpolate

    def forward(self, input):
        x=self.backbone(input)
        x = self.conv1(x)
        x= self.bn1(x)
        x = self.relu(x)
        x = self.conv2(x)
        x= self.bn2(x)
        x = self.relu(x)
        x = F.interpolate(x, size=[512,512], mode='bilinear', align_corners=True)
        return x

The part of training

    self.criterion=nn.CrossEntropyLoss()
    if self.args.cuda:
        image, target = image.cuda(), target.cuda()
    self.scheduler(self.optimizer, i, epoch, self.best_pred)
    self.optimizer.zero_grad()
    output = self.model(image)
    loss = self.criterion(output, target.long())
    loss.backward()

But the Error occurs

File "E:/python_workspace/1006/train.py", line 135, in training
loss = self.criterion(output, target.long())
File "E:\python_workspace\1006\utils\loss.py", line 28, in CrossEntropyLoss
loss = criterion(logit, target.long())
File "E:\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "E:\anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 916, in forward
ignore_index=self.ignore_index, reduction=self.reduction)
File "E:\anaconda3\lib\site-packages\torch\nn\functional.py", line 1995, in cross_entropy
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
File "E:\anaconda3\lib\site-packages\torch\nn\functional.py", line 1826, in nll_loss
ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at C:\w\1\s\tmp_conda_3.6_045031\conda\conda-bld\pytorch_1565412750030\work\aten\src\THNN/generic/SpatialClassNLLCriterion.c:111

image.shape is [4, 3, 512, 512],dtype is torch.float32
target.shape is [4, 512, 512],dtype is torch.float32
output.shape is [4, 3, 512, 512],dtype is torch.float32

target image
The target images all only have three different colors.so I set output to 3 channel.And there's Image mode is P
Where may have problems in my code?

Output:
Judging by the sizes of your ternsors, your batch_size=4. You are trying to predict one of three labels per pixel, that is n_classes=3.

The error you got:

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.

Means that the target.long() you provide to your loss function has values either negative or larger than n_classes.

Check the way you read the ground truth labels. If it's a P type image, you need to read it as such and not convert it to RGB values.

PS,
Do not use align_corners=True in F.interpolate, it introduces distortions.


Instruction: What does a.sub_(lr*a.grad) actually do?

Input:
I am doing the course of fast-ai, SGD and I can not understand.....

This subtracts the coefficients by (learning rate * gradient)...

But why is it necessary to subtract?

here is the code:

def update():
  y_hat = x@a
  loss = mse(y_hat, y)
  if t % 10 == 0: print (loss)
  loss.backward()
  with torch.no_grad():
    a.sub_(lr * a.grad)

Output:
Look at the image. It shows the loss function J as a function of the parameter W. Here it is a simplified representation with W being the only parameter. So, for a convex loss function, the curve looks as shown.

Note that the learning rate is positive. On the left side, the gradient (slope of the line tangent to the curve at that point) is negative, so the product of the learning rate and gradient is negative. Thus, subtracting the product from W will actually increase W (since 2 negatives make a positive). In this case, this is good because loss decreases.

On the other hand (on the right side), the gradient is positive, so the product of the learning rate and gradient is positive. Thus, subtracting the product from W reduces W. In this case also, this is good because the loss decreases.

We can extend this same thing for more number of parameters (the graph shown will be higher dimensional and won't be easy to visualize, which is why we had taken a single parameter W initially) and for other loss functions (even non-convex ones, though it won't always converge to the global minima, but definitely to the nearest local minima).

Note : This explanation can be found in Andrew Ng's courses of deeplearning.ai, but I couldn't find a direct link, so I wrote this answer.


Instruction: Learning rate & gradient descent difference?

Input:
What is the difference between the two?, the two serve to reach the minimum point (lower loss) of a function for example.

I understand (I think)  that the learning rate is multiplied by the gradient ( slope ) to make the  gradient descent , but is that so ? Do I miss something?

What is the difference between lr and gradient?

Thanks

Output:
Deep learning neural networks are trained using the stochastic gradient descent algorithm.

Stochastic gradient descent is an optimization algorithm that estimates the error gradient for the current state of the model using examples from the training dataset, then updates the weights of the model using the back-propagation of errors algorithm, referred to as simply backpropagation.

The amount that the weights are updated during training is referred to as the step size or the “learning rate.”

  Specifically, the learning rate is a configurable hyperparameter
  used in the training of neural networks that has a small positive
  value, often in the range between 0.0 and 1.0.

The learning rate controls how quickly the model is adapted to the problem. Smaller learning rates require more training epochs given the smaller changes made to the weights each update, whereas larger learning rates result in rapid changes and require fewer training epochs.

A learning rate that is too large can cause the model to converge too quickly to a suboptimal solution, whereas a learning rate that is too small can cause the process to get stuck.

The challenge of training deep learning neural networks involves carefully selecting the learning rate. It may be the most important hyperparameter for the model.

The learning rate is perhaps the most important hyperparameter. If you have time to tune only one hyperparameter, tune the learning rate.

  — Page 429, Deep Learning, 2016.

For more on what the learning rate is and how it works, see the post:

How to Configure the Learning Rate Hyperparameter When Training Deep Learning Neural Networks

Also you can refer to here: Understand the Impact of Learning Rate on Neural Network Performance


Instruction: What is the meaning of a 'mini-batch' in deep learning?

Input:
I'm taking the fast-ai course, and in "Lesson 2 - SGD" it says:

Mini-batch: a random bunch of points that you use to update your weights

And it also says that gradient descent uses mini-batches.
What is a mini-batch?  What's the difference between a mini-batch and a regular batch?

Output:
Both are approaches to gradient descent. But in a batch gradient descent you process the entire training set in one iteration. Whereas, in a mini-batch gradient descent you process a small subset of the training set in each iteration.

Also compare stochastic gradient descent, where you process a single example from the training set in each iteration.

Another way to look at it: they are all examples of the same approach to  gradient descent with a batch size of m and a training set of size n. For stochastic gradient descent, m=1. For batch gradient descent, m = n. For mini-batch, m=b and b < n, typically b is small compared to n.

Mini-batch adds the question of determining the right size for b, but finding the right b may greatly improve your results.


Instruction: In distributed computing, what are world size and rank?

Input:
I've been reading through some documentation and example code with the end goal of writing scripts for distributed computing (running PyTorch), but the concepts confuse me.

Let's assume that we have a single node with 4 GPUs, and we want to run our script on those 4 GPUs (i.e. one process per GPU). In such a scenario, what are the rank world size and rank? I often find the explanation for world size: Total number of processes involved in the job, so I assume that that is four in our example, but what about rank?

To explain it further, another example with multiple nodes and multiple GPUs could be useful, too.

Output:
These concepts are related to parallel computing. It would be helpful to learn a little about parallel computing, e.g., MPI.
You can think of world as a group containing all the processes for your distributed training. Usually, each GPU corresponds to one process. Processes in the world can communicate with each other, which is why you can train your model distributedly and still get the correct gradient update. So world size is the number of processes for your training, which is usually the number of GPUs you are using for distributed training.
Rank is the unique ID given to a process, so that other processes know how to identify a particular process. Local rank is the a unique local ID for processes running in a single node, this is where my view differs with @zihaozhihao.
Let's take a concrete example. Suppose we run our training in 2 servers (some articles also call them nodes) and each server/node has 4 GPUs. The world size is 4*2=8. The ranks for the processes will be [0, 1, 2, 3, 4, 5, 6, 7]. In each node, the local rank will be [0, 1, 2, 3].
I have also written a post about MPI collectives and basic concepts. The link is here.


Instruction: Problem when converting Pytorch Image Classifier to mlmodel: Returns same softmax output regardless of img

Input:
I trained and tested an image classifier (Resnet34, Fast.ai, 3 classes) using pytorch and learn.predict() works as expected. When I convert pytorch -> onnx -> mlmodel it predicts the same softmax values regardless of the image I submit.

Here's my pytorch model:

Sequential(
  (0): Sequential(
    (0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
    (3): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (4): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (5): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (3): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (6): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (3): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (4): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (5): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (7): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
  )
  (1): Sequential(
    (0): AdaptiveConcatPool2d(
      (ap): AdaptiveAvgPool2d(output_size=1)
      (mp): AdaptiveMaxPool2d(output_size=1)
    )
    (1): Flatten()
    (2): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (3): Dropout(p=0.25, inplace=False)
    (4): Linear(in_features=1024, out_features=512, bias=True)
    (5): ReLU(inplace=True)
    (6): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (7): Dropout(p=0.5, inplace=False)
    (8): Linear(in_features=512, out_features=3, bias=True)
  )
)

To convert it to .onnx, I need to first normalize the image data and flatten it. I found this tutorial, which worked on a previous version of fastai/onnx-coreml. I do this with the following class:

sz = (960,540)
class ImageScale(nn.Module):
    def __init__(self):
        super().__init__()
        self.denominator = torch.full((3, sz[0], sz[1]), 255.0, device=torch.device("cuda"))

    def forward(self, x): return torch.div(x, self.denominator).unsqueeze(0)

To construct the entire model, I concatenate my ImageScale layer, the model, and a softmax function like this:

final_model = [ImageScale()] + [learn.model] + [nn.Softmax(dim=-1)]
final_model = nn.Sequential(*final_model)

Which ends up looking like this:

Sequential(
  (0): ImageScale()
  (1): Sequential(
    (0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
    (3): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (4): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (5): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (3): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (6): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (3): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (4): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (5): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (7): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
  )
  (2): Sequential(
    (0): AdaptiveConcatPool2d(
      (ap): AdaptiveAvgPool2d(output_size=1)
      (mp): AdaptiveMaxPool2d(output_size=1)
    )
    (1): Flatten()
    (2): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (3): Dropout(p=0.25, inplace=False)
    (4): Linear(in_features=1024, out_features=512, bias=True)
    (5): ReLU(inplace=True)
    (6): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (7): Dropout(p=0.5, inplace=False)
    (8): Linear(in_features=512, out_features=3, bias=True)
  )
  (3): Softmax(dim=-1)
)

I convert to .onnx like this:

dummy_input = Variable(torch.randn(3, sz[0], sz[1])).cuda()
torch.onnx.export(final_model, dummy_input, 'model.onnx', input_names = ['input'], output_names =['output'], verbose=True)

And I convert from .onnx to .mlmodel like this:

model_file = open('model.onnx', 'rb')
model_proto = onnx_pb.ModelProto()
model_proto.ParseFromString(model_file.read())
coreml_model = convert(model_proto, image_input_names = ['image'], mode='classifier', class_labels="labels.txt")
coreml_model.save('model.mlmodel')

When I call predict using coremltools, I get the same output regardless of the image I input:

import coremltools
from PIL import Image
model =  coremltools.models.MLModel('model.mlmodel')
img = Image.open('img.jpg')
preds = model.predict({'image': img})
# preds: {'output': {'class1': 0.011085365898907185, 'class2': 0.9794686436653137, 'class2': 0.009446004405617714}, 'classLabel': 'class2'}
img2 = Image.open('img2.jpg')
preds = model.predict({'image': img2})
# preds: {'output': {'class1': 0.011085365898907185, 'class2': 0.9794686436653137, 'class2': 0.009446004405617714}, 'classLabel': 'class2'}

Possible issues:
1. I'm not setting up the Sequential correctly before converting
2. Coreml or onnx cannot handle non-square images

I've tried a bunch of different inputs, but keep getting the same so any help would be much appreciated!

Here are screen shots of my head and tail from netron:

Head:

Tail:

Output:
I added target_ios=13 to my list of parameters (which required updating to MacOS Version 10.15) and it worked.

from onnx_coreml import convert
ml_model = convert(model='model.onnx', target_ios='13')


Instruction: AttributeError: 'list' object has no attribute 'dim' when predicting in pytorch

Input:
I'm currently loading in a model and 11 input values. Then I'm sending those 11 values into a tensor and attempting to predict outputs.
Here is my code:

# coding: utf-8

# In[5]:

import torch
import torchvision
from torchvision import transforms, datasets
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as utils
import numpy as np

data_np = np.loadtxt('input_preds.csv', delimiter=',')

train_ds = utils.TensorDataset(torch.tensor(data_np, dtype=torch.float32).view(-1,11))

trainset = torch.utils.data.DataLoader(train_ds, batch_size=1, shuffle=True)

# setting device on GPU if available, else CPU, replace .cuda() with .to(device)
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        #self.bn = nn.BatchNorm2d(11)
        self.fc1 = nn.Linear(11, 22)
        self.fc2 = nn.Linear(22, 44)
        self.fc3 = nn.Linear(44, 22)
        self.fc4 = nn.Linear(22, 11)

    def forward(self, x):
        #x = x.view(-1, 11)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        #return F.log_softmax(x, dim=1)
        return x
model1 = torch.load('./1e-2')
model2 = torch.load('./1e-3')

for data in trainset:
        X = data
        X = X

        output = model1(X).to(device)
        print(output)

However, I get this error

Traceback (most recent call last):
  File "inference.py", line 53, in <module>
    output = model1(X).to(device)
  File "C:\Users\Happy\Miniconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "inference.py", line 40, in forward
    x = F.relu(self.fc1(x))
  File "C:\Users\Happy\Miniconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Happy\Miniconda3\envs\torch\lib\site-packages\torch\nn\modules\linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\Happy\Miniconda3\envs\torch\lib\site-packages\torch\nn\functional.py", line 1022, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'list' object has no attribute 'dim'

I've tried to convert the batch to a numpy array but that didn't help. How do I resolve this error? Thank you for your help.

Output:
It looks like your X (data) is a list of tensors, while a PyTorch tensor is expected.
Try X = torch.stack(X).to(device) before sending to the model.


Instruction: Word embeddings with multiple categorial features for a single word

Input:
I'm looking for a method to implement word embedding network with LSTM layers in Pytorch such that the input to the nn.Embedding layer has a different form than vectors of words IDs.

Each word in my case has a corresponding vector and the sentence in my corpus is consequently a vector of vectors. So, for example, I may have the word "King" with vector [500, 3, 18] where 500 is the Word ID, 3 is the word color, and 18 is the font size, etc. The embedding layer role here is to do some automatic feature reduction/extraction.

How can I feed the embedding layer with such form data? Or do you have any better suggestions?

Output:
I am not sure what do you mean by word2vec algorithm with LSTM because the original word2vec algorithm does not use LSTMs and uses directly embeddings to predict surrounding words.

Anyway, it seems you have multiple categorical variables to embed. In the example, it is word ID, color ID, and font size (if you round it to integer values). You have two option:

You can create new IDs for all possible combinations of your features and use nn.Embedding for them. There is however a risk that most of the IDs will appear too sparsely in the data to learn reliable embeddings.
Have separate embedding for each of the features. Then, you will need to combine the embeddings for the features together. You have basically three options how to do it:

Just concatenate the embeddings and let the following layers of the network to resolve the combination.
Choose the same embedding dimension for all features and average them. (I would start with this one probably.)
Add a nn.Dense layer (or two, the first one with ReLU activation and the second without activation) that will explicitly combine the embeddings for your features.

If you need to include continuous features that cannot be discretized,  you can always take the continuous features, apply a layer or two on top of them and combine them with the embeddings of the discrete features.


Instruction: What's the workaround for "ragged/jagged tensors" in PyTorch?

Input:
Tensorflow provides ragged tensors (https://www.tensorflow.org/guide/ragged_tensor). PyTorch however doesn't provide such a data structure. Is there a workaround to construct something similar in PyTorch?

import numpy as np
x = np.array([[0], [0, 1]])
print(x)  # [list([0]) list([0, 1])]

import tensorflow as tf
x = tf.ragged.constant([[0], [0, 1]])
print(x)  # <tf.RaggedTensor [[0], [0, 1]]>

import torch
# x = torch.Tensor([[0], [0, 1]])  # ValueError

Output: PyTorch is implementing something called NestedTensors which seems to have pretty much the same purpose as RaggedTensors in Tensorflow. You can follow the RFC and progress here.


Instruction: Convert 3D Tensor to 4D Tensor in Pytorch

Input:
I had difficulty finding information on reshaping in PyTorch. Tensorflow is quite easy.

My tensor has shape torch.Size([3, 480, 480]).
I want to convert it to a 4D tensor with shape [1,3,480,480].
How do I do that?

Output:
You can use unsqueeze()

For example:

x = torch.zeros((4,4,4))   # Create 3D tensor
x = x.unsqueeze(0)         # Add dimension as the first axis (1,4,4,4)

I've seen a few people use indexing with None to add a singular dimension as well. For example:

x = torch.zeros((4,4,4))   # Create 3D tensor
print(x[None].shape)       #  (1,4,4,4)
print(x[:,None,:,:].shape) #  (4,1,4,4)
print(x[:,:,None,:].shape) #  (4,4,1,4)
print(x[:,:,:,None].shape) #  (4,4,4,1)

Personally, I prefer unsqueeze(), but it's good to be familiar with both.