Instruction: Freezing layers in pre-trained bert model

Input: How to freeze the last two layers in the above pre-trained model (dropout and classifier layers)? So that when the model is run, I will get a dense layer as output.

Output:
I would like to point you to the definition of BertForSequenceClassification and you can easily avoid the dropout and classifier by using:

model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
model.bert() # this will give you the dense layer output

Why you can do the above? If you take a look at the constructor of BertForSequenceClassification:

def __init__(self, config):
    super(BertForSequenceClassification, self).__init__(config)
    self.num_labels = config.num_labels

    self.bert = BertModel(config)
    self.dropout = nn.Dropout(config.hidden_dropout_prob)
    self.classifier = nn.Linear(config.hidden_size, self.config.num_labels)

    self.init_weights()

As you can see, you just want to ignore the dropout and classifier layers.

One more thing, freezing a layer and removing a layer are two different things. In your question, you mentioned that you want to freeze the classifier layer but freezing a layer will not help you to avoid it. Freezing means, you do not want to train the layer.


Instruction: Is .data still useful in pytorch 1.3 stable and what is the meaning of it?

Input:
Is .data still be used in the pytorch 1.3 stable, if so, could you please share the reference to me? Thx.

t = torch.randperm(8)
t.data

Output:
From PyTorch v0.4.0, calling y = x.data still has similar semantics. So y will be a Tensor that shares the same data with x, is unrelated to the computation history of x, and has requires_grad=False.

However, .data can be unsafe in some cases. Any changes on x.data wouldn't be tracked by autograd, and the computed gradients would be incorrect if x is needed in a backward pass. A safer alternative is to use x.detach(), which also returns a Tensor that shares data with requires_grad=False, but will have its in-place changes reported by autograd if x is needed in backward.

Reference: https://github.com/pytorch/pytorch/releases/tag/v0.4.0


Instruction: Dimension mismatch while using Pytorch LSTM module

Input:
I have a pytorch pretrained model from which I am generating features/embedding for some input sentences. The features are essentially torch object. For example a sample input_embedding (a list of torch objects) for one sentence looks like below

[tensor([-0.8264,  0.2524], device='cuda:0', grad_fn=<SelectBackward>)]

Now, I want to pass this embedding through a custom model which is fundamentally a bi-directional LSTM:

def custom_model(input_embedding):
    #initialize BiLSTM
    bilstm = torch.nn.LSTM(input_size=1, hidden_size=1, num_layers=1, batch_first=False, bidirectional=True)
    #feed input to bilstm object
    bi_output, bi_hidden = bilstm(input_embedding)
    # more code ....
    return F.softmax(x)

I wanted to pass my input_embedding to this custom model to get the prediction output like below:

for item in input_embedding:
    y_pred = biLSTM_single_sentence_student_model(item)

But it is throwing error on the bi_output, bi_hidden = bilstm(input_embedding) line saying:

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Most likely I am not defining the bilstm object properly due to my lack of understanding of Pytorch nn.LSTM input.

Please suggest.

Output:
The input to LSTM must be a 3d tensor of shape (seq_len, batch, input_size). In your example, you are basically providing a 2d tensor of shape seq_len, input_size as you mentioned, [-0.8264,  0.2524] is one example sentence.

So, you can modify your example as follows.

# a list of 2 sentences
input_embedding = [
    torch.FloatTensor([[-0.8264],  [0.2524]]),
    torch.FloatTensor([[-0.3259],  [0.3564]])
]
for item in input_embedding:
    # item is a 2d tensor of shape `seq_len, input_size`
    # so, we unsqueeze it to make it 3d `seq_len, 1, input_size` where batch_size = 1
    y_pred = custom_model(item.unsqueeze(1))
    print(y_pred.size()) # 2, 1, 2

Hope this helps!


Instruction: Extracting fixed vectors from BioBERT without using terminal command?

Input:
If we want to use weights from pretrained BioBERT model, we can execute following terminal command after downloading all the required BioBERT files.

os.system('python3 extract_features.py \
      --input_file=trial.txt \
      --vocab_file=vocab.txt \
      --bert_config_file=bert_config.json \
      --init_checkpoint=biobert_model.ckpt \
      --output_file=output.json')

The above command actually reads individual file containing the text, reads the textual content from it, and then writes the extracted vectors to another file. So, the problem with this is that it could not be scaled easily for very large data-sets containing thousands of sentences/paragraphs.

Is there is a way to extract these features on the go (using an embedding layer) like it could be done for the word2vec vectors in PyTorch or TF1.3?

Note: BioBERT checkpoints do not exist for TF2.0, so I guess there is no way it could be done with TF2.0 unless someone generates TF2.0 compatible checkpoint files.

I will be grateful for any hint or help.

Output:
You can get the contextual embeddings on the fly, but the total time spend on getting the embeddings will always be the same. There are two options how to do it: 1. import BioBERT into the Transformers package and treat use it in PyTorch (which I would do) or 2. use the original codebase.

1. Import BioBERT into the Transformers package

The most convenient way of using pre-trained BERT models is the Transformers package. It was primarily written for PyTorch, but works also with TensorFlow. It does not have BioBERT out of the box, so you need to convert it from TensorFlow format yourself. There is convert_tf_checkpoint_to_pytorch.py script that does that. People had some issues with this script and BioBERT (seems to be resolved).

After you convert the model, you can load it like this.

import torch
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('directory_with_converted_model')
model = BertModel.from_pretrained('directory_with_converted_model')

# Call the model in a standard PyTorch way
embeddings = model([tokenizer.encode("Cool biomedical tetra-hydro-sentence.", add_special_tokens=True)])

2. Use directly BioBERT codebase

You can get the embeddings on the go basically using the code that is exctract_feautres.py. On lines 346-382, they initialize the model. You get the embeddings by calling estimator.predict(...).

For that, you need to format your format the input. First, you need to format the string (using code on line 326-337) and then apply and call convert_examples_to_features on it.


Instruction: Masking tensor of same shape in PyTorch

Input:
Given an array and mask of same shapes, I want the masked output of the same shape and containing 0 where mask is False.

For example,

# input array
img = torch.randn(2, 2)
print(img)
# tensor([[0.4684, 0.8316],
#        [0.8635, 0.4228]])
print(img.shape)
# torch.Size([2, 2])

# mask
mask = torch.BoolTensor(2, 2)
print(mask)
# tensor([[False,  True],
#        [ True,  True]])
print(mask.shape)
# torch.Size([2, 2])

# expected masked output of shape 2x2
# tensor([[0, 0.8316],
#        [0.8635, 0.4228]])

Issue: The masking changes the shape of the output as follows:

#1: shape changed
img[mask]
# tensor([0.8316, 0.8635, 0.4228])

Output:
Simply type-cast your boolean mask to an integer mask, followed by float to bring the mask to the same type as in img. Perform element-wise multiplication afterwards.

masked_output = img * mask.int().float()


Instruction: Select/Mask different column index in every row

Input:
In pytorch I have a multi-dimensional tensor, call it X

X = [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], ...]

Now I would like to select a different column index for each row like so

indices = [[0], [1], [0], [2], ...]
# now I expect following values to be returned:
[[1], [5], [7], [12], ...]

also I would like to achieve the opposite so that for the given indices I get

[[2, 3], [4, 6], [8, 9], [10, 11]]

Is there a "simple" way to achieve this without a for loop? I would be grateful for any ideas.

Output:
In fact the torch.gather function performs exactly this.

For example

a = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
indices = torch.tensor([[0], [1], [0], [2]])
a.gather(1, indices)

will exactly return

tensor([[ 1],
        [ 5],
        [ 7],
        [12]])

I do not require the opposite anymore but for this I would propose just creating a mask with all ones and then setting the respective indices of the "gathering" tensor to 0 or just create a new "gathering" tensor which contains the respective opposite keys. For example:

indices_opposite = [np.setdiff1d(np.arange(a.size(1)), i) for i in indices.numpy()]


Instruction: How to train features in different scales in deep learning model

Input:
I'm new in deep learning and I built a very simple model to try to train my data. I have two features input: sex and age. sex is 0 or 1 and age is between 25 and 60. Output is just 0 means this person has no such disease and 1 means has such disease.

However, when I train my model, the training loss does not decrease at all. It looks like because my two features are very different in range. So how can I fix this? Any suggestions would be greatly appreciated.

My code is here:

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()

        self.fc1 = nn.Sequential(
            nn.Linear(2,50),
            nn.ReLU(),
            nn.Linear(50,2)
        )

    def forward(self,x):
        x = self.fc1(x)

        x = F.softmax(x, dim=1)
        return x

#Inputs
X = np.column_stack((sex,age))
X = torch.from_numpy(X).type(torch.FloatTensor)
y = torch.from_numpy(y).type(torch.LongTensor)

#Initialize the model
model = Net()
#Define loss criterion
criterion = nn.CrossEntropyLoss()
#Define the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

epochs = 1000

losses = []
for i in range(epochs):

    y_pred = model.forward(X)
    #Compute Cross entropy loss
    loss = criterion(y_pred,y)
    #Add loss to the list
    losses.append(loss.item())
    #Clear the previous gradients
    optimizer.zero_grad()
    #Compute gradients
    loss.backward()
    #Adjust weights
    optimizer.step()
    _, predicted = torch.max(y_pred.data, 1)

    if i % 50 == 0:
        print(loss.item())

And the train loss looks like this

0.9273738861083984
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618

EDIT

Thank you for your comments. Sorry I didn't explain my question clearly. This is part of my network and my input data contains two parts: the first part is some signal data and I use CNN model to train it and it works well; the second part is what I mentioned above. My goal is to merge two models to improve my accuracy.
I've tried normalization and looks like it works. I want to know is it always necessary to do normalization when pre-processing data? Thank you!

Output:
An alternative.

If the age has discrete values in the range (25-60), then one possible way would be to learn embeddings for those two attributes, sex and age.

For example,

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.sex_embed = nn.Embedding(2, 20)
        self.age_embed = nn.Embedding(36, 50)

        self.fc1 = nn.Sequential(
            nn.Linear(70, 35),
            nn.ReLU(),
            nn.Linear(35, 2)
        )

    def forward(self, x):
        # write the forward

In the above example, I assume age values will be integer (25, 26, ..., 60), so for each possible value, we can learn a vector representation.

So, I propose to learn a 20d representation of sex and 50d representation of age. You can change the dimensions and do experiments to find an optimal value.


Instruction: How can I shuffle the labels of a dataset?

Input:
I have downloaded the MNIST dataset, using the following command:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

I now need to run some experiments on this dataset (MNIST), but shuffling the labels of the training set. How can I shuffle/reassign them randomly? I have tried the following:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            target_transform=lambda y: torch.randint(0, 10, (1,)).item(),
                            download=True)

But I have noticed that what comes after the lambda function makes the labels shuffle during the training process, e.g. they change at every epoch. This way, I won't reach 100% training accuracy, which is what I am aiming for. How can I shuffle these labels in  a way that is completely random, making sure that these labels won't change during the training process?

Thank you!!

Output:
In case your goal is to create a random mapping of labels you would need to define the mapping before defining the target transform to keep the transform constant. Something like the following should do the trick

import random
label_mapping = list(range(10))
random.shuffle(label_mapping)
train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            target_transform=lambda y: label_mapping[y],
                            download=True)

In order to get a new shuffle each epoch you would want to redefine the label mapping, training dataset, and dataloader each epoch.

Update To instead generate a random label which is independent of the true label but consistent for a given index then you probably need to either do some very careful seeding or reimplement some functionality of the dataset class.

For example, the latter case might look something like this

import random
class RandomMNIST(dsets.MNIST):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.targets = [random.randint(0, 9) for _ in range(len(self.data))]

train_dataset = RandomMNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

or equivalently

import random
train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)
train_dataset.targets = [random.randint(0, 9) for _ in range(len(train_dataset))]


Instruction: Deleting Rows in Torch Tensor

Input:
I have a torch tensor as follows -

a = tensor(
[[0.2215, 0.5859, 0.4782, 0.7411],
[0.3078, 0.3854, 0.3981, 0.5200],
[0.1363, 0.4060, 0.2030, 0.4940],
[0.1640, 0.6025, 0.2267, 0.7036],
[0.2445, 0.3032, 0.3300, 0.4253]],  dtype=torch.float64)

If the first value of each row is less than 0.2 then the whole row needs to be deleted. Thus I need the output like -

tensor(
[[0.2215, 0.5859, 0.4782, 0.7411],
[0.3078, 0.3854, 0.3981, 0.5200],
[0.2445, 0.3032, 0.3300, 0.4253]],  dtype=torch.float64)

I have tried to loop through the tensor and append the valid value to a new empty tensor but was not successful. Is there any way to get the results efficiently?

Output:
Code

a = torch.Tensor(
    [[0.2215, 0.5859, 0.4782, 0.7411],
    [0.3078, 0.3854, 0.3981, 0.5200],
    [0.1363, 0.4060, 0.2030, 0.4940],
    [0.1640, 0.6025, 0.2267, 0.7036],
    [0.2445, 0.3032, 0.3300, 0.4253]])

y = a[a[:, 0] > 0.2]
print(y)

Output

tensor([[0.2215, 0.5859, 0.4782, 0.7411],
        [0.3078, 0.3854, 0.3981, 0.5200],
        [0.2445, 0.3032, 0.3300, 0.4253]])


Instruction: Why is the input size of the MultiheadAttention in Pytorch Transformer module 1536?

Input:
When using the torch.nn.modules.transformer.Transformer module/object, the first layer is the encoder.layers.0.self_attn layer that is a MultiheadAttention layer, i.e.

from torch.nn.modules.transformer import Transformer
bumblebee = Transformer()

bumblee.parameters

[out]:

<bound method Module.parameters of Transformer(
  (encoder): TransformerEncoder(
    (layers): ModuleList(
      (0): TransformerEncoderLayer(
        (self_attn): MultiheadAttention(
          (out_proj): Linear(in_features=512, out_features=512, bias=True)
        )
        (linear1): Linear(in_features=512, out_features=2048, bias=True)
        (dropout): Dropout(p=0.1, inplace=False)
        (linear2): Linear(in_features=2048, out_features=512, bias=True)
        (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        (dropout1): Dropout(p=0.1, inplace=False)
        (dropout2): Dropout(p=0.1, inplace=False)
      )

And if we print out the size of the layer, we see:

for name in bumblebee.encoder.state_dict():
    print(name, '\t', bumblebee.encoder.state_dict()[name].shape)

[out]:

layers.0.self_attn.in_proj_weight    torch.Size([1536, 512])
layers.0.self_attn.in_proj_bias      torch.Size([1536])
layers.0.self_attn.out_proj.weight   torch.Size([512, 512])
layers.0.self_attn.out_proj.bias     torch.Size([512])
layers.0.linear1.weight      torch.Size([2048, 512])
layers.0.linear1.bias    torch.Size([2048])
layers.0.linear2.weight      torch.Size([512, 2048])
layers.0.linear2.bias    torch.Size([512])
layers.0.norm1.weight    torch.Size([512])
layers.0.norm1.bias      torch.Size([512])
layers.0.norm2.weight    torch.Size([512])
layers.0.norm2.bias      torch.Size([512])

It seems like 1536 is 512 * 3 and somehow the layers.0.self_attn.in_proj_weight parameter might be storing all three QKV tensors in the transformer architecture in one matrix.

From https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/activation.py#L649

class MultiheadAttention(Module):
    def __init__(self, embed_dim, num_heads, dropout=0., bias=True, add_bias_kv=False, add_zero_attn=False, kdim=None, vdim=None):
        super(MultiheadAttention, self).__init__()
        self.embed_dim = embed_dim
        self.kdim = kdim if kdim is not None else embed_dim
        self.vdim = vdim if vdim is not None else embed_dim
        self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim

        self.num_heads = num_heads
        self.dropout = dropout
        self.head_dim = embed_dim // num_heads
        assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"

        if self._qkv_same_embed_dim is False:
            self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim))
            self.k_proj_weight = Parameter(torch.Tensor(embed_dim, self.kdim))
            self.v_proj_weight = Parameter(torch.Tensor(embed_dim, self.vdim))
        else:
            self.in_proj_weight = Parameter(torch.empty(3 * embed_dim, embed_dim))

And the note in the docstring of the MultiheadAttention says:

  Note: if kdim and vdim are None, they will be set to embed_dim such that
  query, key, and value have the same number of features.

Is that correct?

Output:
From the nn.Transformer definition with the default values, EncoderLayer is instantiated with d_model=512, nhead=8.

The MultiheadAttention is instantiated with d_model, nhead equal to those values and k_dim, v_dim are left to the default value of None.

If they are None, self._qkv_same_embed_dim at this line evaluates to True. When that happens, as you correctly pointed out self.in_proj_weight is defined as a Tensor of shape (3 x embed_dim, embed_dim).

In short: yes, that's correct.


Instruction: PyTorch, apply different functions element-wise

Input:
I defined a tensor like this

t_shape = [4, 1]
data = torch.rand(t_shape)

I want to apply different functions to each row.

funcs = [lambda x: x+1, lambda x: x**2, lambda x: x-1, lambda x: x*2]  # each function for each row.

I can do it with the following code

d = torch.tensor([f(data[i]) for i, f in enumerate(funcs)])

How can I do it in a proper way with more advanced APIs defined in PyTorch?

Output:
I think your solution is good. But it won't work with any tensor shape. You can slightly modify the solution as follows.

t_shape = [4, 10, 10]
data = torch.rand(t_shape)

funcs = [lambda x: x+1, lambda x: x**2, lambda x: x-1, lambda x: x*2]

# only change the following 2 lines
d = [f(data[i]) for i, f in enumerate(funcs)]
d = torch.stack(d, dim=0)


Instruction: Can I train two networks, one network which contains another netwrok, at the same time?

Input:
I define two networks in my model.

def Net_One():
    conv2d
    conv2d
    ...
def Net_Two():
    Net_One(input)
    conv2d
    fc

So my question is: when I train Net_Two, using back propagation optimization, will pytorch automatically train Net_One or not? Why?

Output:
Firstly model class name's convention should be CapWords: NetOne,NetTwo, but this doesn't hurt anything, just a convention.

As for your questions, It depends on the processing of NetTwo().
if the final loss of NetTwo has nothing to do with NetOne(), then the back propagation will not flow through NetOne(), and therefore will not update NetOne()'s parameters.
Otherwise the back propagation will compute the NetOne()'s gradients and update its weights.

For code samples:

# NetTwo's loss has nothing to do with NetOne:
def NetOne():
    def __init__(self):
        super(NetOne, self).__init__()
        ...

    def forward(inputs):
        ...

def NetTwo():
    def __init__(self):
        super(NetTwo, self).__init__()
        ...

    def forward(inputs):
        # in this processing, temp is never used by NetTwo's layers..
        ...
        temp = NetOne(inputs)
        inputs = conv2d(inputs)
        ...

In the upper codes, temp is never used, so NetOne won't be updated when NetTwo is updating.

But if temp is used by NetTwo.forward(), it'll be updated, like following:

def NetTwo():
    def __init__(self):
        super(NetTwo, self).__init__()
        ...

    def forward(inputs):
        # in this processing, temp is used by NetTwo's layers.
        ...
        temp = NetOne(inputs)
        inputs = conv2d(temp)
        ...

Did this answer your question?


Instruction: How do you efficiently sum the occurences of a value in one array at positions in another array

Input:
Im looking for an efficient 'for loop' avoiding solution that solves an array related problem I'm having. I want to use a huge 1Darray (A -> size = 250.000) of values between 0 and 40 for indexing in one dimension, and a array (B) with the same size with values between 0 and 9995 for indexing in a second dimension.

The result should be an array with size (41, 9996) with for each index the amount of times that any value from array 1 occurs at a value from array 2.

Example:

A = [0, 3, 2, 4, 3]
B = [1, 2, 2, 0, 2]
which should result in:
[[0, 1, 0,
 [0, 0, 0,
 [0, 0, 1,
 [0, 0, 2,
 [1, 0, 0]]

The dirty way is too slow as the amount of data is huge, what you would be able to do is:

out = np.zeros(41,9995)
for i in A:
  for j in B:
     out[i,j] += 1

which will take 238.000 * 238.000 loops...
I've tried this, which works partially:

out = np.zeros(41,9995)
out[A,B] += 1

Which generates a result with 1 everywhere, regardless of the amount of times the values occur.

Does anyone have a clue how to fix this? Thanks in advance!

Output:
You are looking for a sparse tensor:

import torch

A = [0, 3, 2, 4, 3]
B = [1, 2, 2, 0, 2]
idx = torch.LongTensor([A, B])
torch.sparse.FloatTensor(idx, torch.ones(idx.shape[1]), torch.Size([5,3])).to_dense()

Output:

tensor([[0., 1., 0.],
        [0., 0., 0.],
        [0., 0., 1.],
        [0., 0., 2.],
        [1., 0., 0.]])

You can also do the same with scipy sparse matrix:

import numpy as np
from scipy.sparse import coo_matrix

coo_matrix((np.ones(len(A)), (np.array(A), np.array(B))), shape=(5,3)).toarray()

output:

array([[0., 1., 0.],
       [0., 0., 0.],
       [0., 0., 1.],
       [0., 0., 2.],
       [1., 0., 0.]])

Sometimes it is better to leave the matrix in its sparse representation, rather than forcing it to be "dense" again.


Instruction: How to use BERT just for ENTITY extraction from a Sequence without classification in the NER task?

Input:
My requirement here is given a sentence(sequence), I would like to just extract the entities present in the sequence without classifying them to a type in the NER task. I see that BertForTokenClassification for NER does the classification. Can this be adapted for just the extraction?

Can BERT just be used to do entity extraction/identification?

Output:
Regardless BERT, NER tagging is usually done by tagging with the IOB format (inside, outside, beginning) or something similar (often the end is also explicitly tagged). The inside and beggining tags contain the entity type. Something like this:

Alex B-PER
is O
going O
to O
Los B-LOC
Angeles I-LOC

If you modify your training data, such that there will be only one entity type, the model will only learn to detect the entities without knowing what type the entity is.

Alex B
is O
going O
to O
Los B
Angeles I


Instruction: Custom LSTM model in Pytorch showing input size mismatch

Input:
I have a custom bidirectional LSTM model where the custom part is

- extract the forward and backward last hidden state
- concat those states
- create a fully connected layer and pass it through softmax layer.

The code looks like below:

class customModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(customModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bilstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=False, bidirectional=True)
        self.fcl = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        # Set initial hidden and cell states
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)

        # Forward propagate LSTM
        out, hidden = self.bilstm(x, (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size)
        #concat hidden state of forward and backword
        fw_bilstm = out[-1, :, :self.hidden_size]
        bk_bilstm = out[0, :, :self.hidden_size]
        concat_fw_bw = torch.cat((fw_bilstm, bk_bilstm), dim = 1)
        fc = nn.Linear(concat_fw_bw, num_classes)
        x = F.relu(fc(x))
        return F.softmax(x)

I use below parameters and input

input_size = 2
hidden_size = 32
num_layers = 1
num_classes = 2

input_embedding = [
    torch.FloatTensor([[-0.8264],  [0.2524]]),
    torch.FloatTensor([[-0.3259],  [0.3564]])
]

Then I create a model object

model = customModel(input_size, hidden_size, num_layers, num_classes)

Which then I use like below:

for item in input_embedding:
    print(item.size())
    for epoch in range(1):
        pred = model(item)
        print (pred)

When I run it, I see for this line out, hidden = self.bilstm(x, (h0, c0)), it shows error

RuntimeError: input must have 3 dimensions, got 2

I am not sure why the model is thinking that input must have 3 dimensions when I explicitly specified input_size=2

What am I missing?

Output:
You seem to be missing a (batch or sequence) dimension in your input.

There is a difference between nn.LSTM and nn.LSTMCell. The former -- which is the one you use -- takes whole sequences as inputs. Therefore it needs 3-dimensional inputs of shape (seq_len, batch, input_size).

Let's say you want to give those 4 sequences of letters (which you code as one-hot vectors) as inputs in form of a batch:

x0 = [a,b,c]
x1 = [c,d,e]
x2 = [e,f,g]
x3 = [h,i,j]

### input.size() should give you the following:
(3,4,8)

The seq_len parameter is the size of the sequences: here 3,
The input_size parameter is the size of each input vector: here, the input would be a one-hot vector of size 8,
The batch is the number of sequences you put together: here there are 4 sequences.

NB: It can be easier to grasp by putting the batch sequence first and setting the batch_first as True

Also: if (h_0, c_0) is not provided, both h_0 and c_0 default to zero so it's not useful to create them.


Instruction: After Deleting some Python Files, I can't install pytorch any more via pip

Input:
Unfortunately I deleted some files of Python. I got macOS Catalina and I want to install pytorch with the command:

pip3 install torch

If I enter this in my terminal I'll get

  Building wheel for torch (setup.py) ... error
    ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-wheel-qh35w11o --python-tag cp38
         cwd: /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/
    Complete output (30 lines):
    running bdist_wheel
    running build
    running build_deps
    Traceback (most recent call last):
      File "", line 1, in
      File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 225, in
        setup(name="torch", version="0.1.2.post2",
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/init.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 192, in run
        self.run_command('build')
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'

  ERROR: Failed building wheel for torch
    Running setup.py clean for torch
    ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' clean --all
         cwd: /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch
    Complete output (2 lines):
    running clean
    error: [Errno 2] No such file or directory: '.gitignore'

  ERROR: Failed cleaning build dir for torch
  Failed to build torch
  Installing collected packages: torch
      Running setup.py install for torch ... error
      ERROR: Command errored out with exit status 1:
       command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-record-cerpeh7h/install-record.txt --single-version-externally-managed --compile
           cwd: /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/
      Complete output (23 lines):
      running install
      running build_deps
      Traceback (most recent call last):
        File "", line 1, in
        File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 225, in
          setup(name="torch", version="0.1.2.post2",
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/init.py", line 145, in setup
          return distutils.core.setup(**attrs)
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 99, in run
          self.run_command('build_deps')
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 51, in run
          from tools.nnwrap import generate_wrappers as generate_nn_wrappers
      ModuleNotFoundError: No module named 'tools.nnwrap'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-record-cerpeh7h/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

Is there a way to solve this problem? Do you need further Information?

UPDATE

I changed the way of the enviroment (without anaconda) and so I get this error message:

  ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-zelcwulm/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-zelcwulm/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-record-i1_7akf_/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

Output:
Try running:

pip3 install --upgrade --force-reinstall pip

If that doesn't work you can always try reinstalling python from https://www.python.org/downloads


Instruction: Do all variables in the loss function have to be tensor with grads in pytorch?

Input:
I have the following function

def msfe(ys, ts):
    ys=ys.detach().numpy() #output from the network
    ts=ts.detach().numpy() #Target (true labels)
    pred_class = (ys>=0.5)
    n_0 = sum(ts==0) #Number of true negatives
    n_1 = sum(ts==1) #Number of true positives
    FPE = sum((ts==0)[[bool(p) for p in (pred_class==1)]])/n_0 #False positive error
    FNE = sum((ts==1)[[bool(p) for p in (pred_class==0)]])/n_1 #False negative error
    loss= FPE**2+FNE**2

    loss=torch.tensor(loss,dtype=torch.float64,requires_grad=True)

    return loss

and I wonder, if the autograd in Pytorch works properly, since ys and ts does not have the grad flag.

So my question is: do all the variables (FPE,FNE,ys,ts,n_1,n_0) have to be tensors, before optimizer.step() works, or is it okay that it is only the final function (loss) which is ?

Output:
All of the variables you want to optimise via optimizer.step() need to have gradient.

In your case it would be y predicted by network, so you shouldn't detach it (from graph).

Usually you don't change your targets, so those don't need gradients. You shouldn't have to detach them though, tensors by default don't require gradient and won't be backpropagated.

Loss will have gradient if it's ingredients (at least one) have gradient.

Overall you rarely need to take care of it manually.

BTW. don't use numpy with PyTorch, there is rarely ever the case to do so. You can perform most of the operations you can do on numpy array on PyTorch's tensor.

BTW2. There is no such thing as Variable in pytorch anymore, only tensors which require gradient and those that don't.

Non-differentiability

1.1 Problems with existing code

Indeed, you are using functions which are not differentiable (namely >= and ==). Those will give you trouble only in the case of your outputs, as those required gradient (you can use == and >= for targets though).

Below I have attached your loss function and outlined problems in it in the comments:

# Gradient can't propagate if you detach and work in another framework
# Most Python constructs should be fine, detaching will ruin it though.
def msfe(outputs, targets):
    # outputs=outputs.detach().numpy() # Do not detach, no need to do that
    # targets=targets.detach().numpy() # No need for numpy either
    pred_class = outputs >= 0.5  # This one is non-differentiable
    # n_0 = sum(targets==0) # Do not use sum, there is pytorch function for that
    # n_1 = sum(targets==1)

    n_0 = torch.sum(targets == 0)  # Those are not differentiable, but...
    n_1 = torch.sum(targets == 1)  # It does not matter as those are targets

    # FPE = sum((targets==0)[[bool(p) for p in (pred_class==1)]])/n_0 # Do not use Python bools
    # FNE = sum((targets==1)[[bool(p) for p in (pred_class==0)]])/n_1 # Stay within PyTorch
    # Those two below are non-differentiable due to == sign as well
    FPE = torch.sum((targets == 0.0) * (pred_class == 1.0)).float() / n_0
    FNE = torch.sum((targets == 1.0) * (pred_class == 0.0)).float() / n_1
    # This is obviously fine
    loss = FPE ** 2 + FNE ** 2

    # Loss should be a tensor already, don't do things like that
    # Gradient will not be propagated, you will have a new tensor
    # Always returning gradient of `1` and that's all
    # loss = torch.tensor(loss, dtype=torch.float64, requires_grad=True)

    return loss

1.2 Possible solution

So, you need to get rid of 3 non-differentiable parts. You could in principle try to approximate it with continuous outputs from your network (provided you are using sigmoid as activation). Here is my take:

def msfe_approximation(outputs, targets):
    n_0 = torch.sum(targets == 0)  # Gradient does not flow through it, it's okay
    n_1 = torch.sum(targets == 1)  # Same as above
    FPE = torch.sum((targets == 0) * outputs).float() / n_0
    FNE = torch.sum((targets == 1) * (1 - outputs)).float() / n_1

    return FPE ** 2 + FNE ** 2

Notice that to minimize FPE outputs will try to be zero on the indices where targets are zero. Similarly for FNE, if targets are 1, network will try to output 1 as well.

Notice similarity of this idea to BCELoss (Binary CrossEntropy).

And lastly, example you can run this on, just for sanity check:

if __name__ == "__main__":
    model = torch.nn.Sequential(
        torch.nn.Linear(30, 100),
        torch.nn.ReLU(),
        torch.nn.Linear(100, 200),
        torch.nn.ReLU(),
        torch.nn.Linear(200, 1),
        torch.nn.Sigmoid(),
    )
    optimizer = torch.optim.Adam(model.parameters())
    targets = torch.randint(high=2, size=(64, 1)) # random targets
    inputs = torch.rand(64, 30) # random data
    for _ in range(1000):
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = msfe_approximation(outputs, targets)
        print(loss)
        loss.backward()
        optimizer.step()

    print(((model(inputs) >= 0.5) == targets).float().mean())


Instruction: need help understanding pytorch blitz math notation

Input:
I just came across this notation in the pytorch blitz tutorial and I dont know what the vertical line is

Does anyone have any suggestions on the notation?

Output:
The vertical line means the value of the left side variable given that the right side variable is a particular value. So, your given example means z_i is 27 when x_i is 1.

Basically, it means 'LHS holds given RHS'


Instruction: How to create a completely (uniformly) random dataset on PyTorch

Input:
I need to run some experiments on custom datasets using pytorch. The question is, how can I create a dataset using torch.Dataloader?

I have two lists, one is called Values and has a datapoint tensor at every entry, and the other one is called Labels, that has the corresponding label. What I did is the following:

for i in range(samples):
dataset[i] = [values[i],labels[I]]

So I have a list with datapoint and respective label, and then tried the following:

dataset = torch.tensor(dataset).float()
dataset = torch.utils.data.TensorDataset(dataset)

data_loader = torch.utils.data.DataLoader(dataset=dataset, batch_size=100, shuffle=True, num_workers=4, pin_memory=True)

But, first of all, I get the error "Not a sequence" in the torch.tensor command, and second, I'm not sure this is the right way of creating one. Any suggestion?

Thank you very much!

Output:
You do not need to overload DataLoader, but rather create a Dataset for your data.
For instance,

class MyDataset(Dataset):
  def __init__(self):
    super(MyDataset, self).__init__()
    # do stuff here?
    self.values = values
    self.labels = labels

  def __len__(self):
    return len(self.values)  # number of samples in the dataset

  def __getitem__(self, index):
    return self.values[index], self.labels[index]


Instruction: PyTorch embedding layer raises "expected...cuda...but got...cpu" error

Input:
I'm working on translating a PyTorch model from CPU (where it works) to GPU (where it so far doesn't). The error message (clipped to the important bits) is as follows:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-a7bb230c924c> in <module>
      1 model = FeedforwardTabularModel()
      2 model.cuda()
----> 3 model.fit(X_train_sample.values, y_train_sample.values)

<ipython-input-11-40b1edae7417> in fit(self, X, y)
    100         for epoch in range(self.n_epochs):
    101             for i, (X_batch, y_batch) in enumerate(batches):
--> 102                 y_pred = model(X_batch).squeeze()
    103                 # scheduler.batch_step()  # Disabled due to a bug, see above.
    104                 loss = self.loss_fn(y_pred, y_batch)

[...]

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

Here is the full model definition:

import torch
from torch import nn
import torch.utils.data
# ^ https://discuss.pytorch.org/t/attributeerror-module-torch-utils-has-no-attribute-data/1666

class FeedforwardTabularModel(nn.Module):
    def __init__(self):
        super().__init__()

        self.batch_size = 512
        self.base_lr, self.max_lr = 0.001, 0.003
        self.n_epochs = 5
        self.cat_vars_embedding_vector_lengths = [
            (1115, 80), (7, 4), (3, 3), (12, 6), (31, 10), (2, 2), (25, 10), (26, 10), (4, 3),
            (3, 3), (4, 3), (23, 9), (8, 4), (12, 6), (52, 15), (22, 9), (6, 4), (6, 4), (3, 3),
            (3, 3), (8, 4), (8, 4)
        ]
        self.loss_fn = torch.nn.MSELoss()
        self.score_fn = torch.nn.MSELoss()

        # Layer 1: embeddings.
        self.embeddings = []
        for (in_size, out_size) in self.cat_vars_embedding_vector_lengths:
            emb = nn.Embedding(in_size, out_size)
            self.embeddings.append(emb)

        # Layer 1: dropout.
        self.embedding_dropout = nn.Dropout(0.04)

        # Layer 1: batch normalization (of the continuous variables).
        self.cont_batch_norm = nn.BatchNorm1d(16, eps=1e-05, momentum=0.1)

        # Layers 2 through 9: sequential feedforward model.
        self.seq_model = nn.Sequential(*[
            nn.Linear(in_features=215, out_features=1000, bias=True),
            nn.ReLU(),
            nn.BatchNorm1d(1000, eps=1e-05, momentum=0.1),
            nn.Dropout(p=0.001),
            nn.Linear(in_features=1000, out_features=500, bias=True),
            nn.ReLU(),
            nn.BatchNorm1d(500, eps=1e-05, momentum=0.1),
            nn.Dropout(p=0.01),
            nn.Linear(in_features=500, out_features=1, bias=True)
        ])

    def forward(self, x):
        # Layer 1: embeddings.
        inp_offset = 0
        embedding_subvectors = []
        for emb in self.embeddings:
            index = torch.tensor(inp_offset, dtype=torch.int64).cuda()
            inp = torch.index_select(x, dim=1, index=index).long().cuda()
            out = emb(inp)
            out = out.view(out.shape[2], out.shape[0], 1).squeeze()
            embedding_subvectors.append(out)
            inp_offset += 1
        out_cat = torch.cat(embedding_subvectors)
        out_cat = out_cat.view(out_cat.shape[::-1])

        # Layer 1: dropout.
        out_cat = self.embedding_dropout(out_cat)

        # Layer 1: batch normalization (of the continuous variables).
        out_cont = self.cont_batch_norm(x[:, inp_offset:])

        out = torch.cat((out_cat, out_cont), dim=1)

        # Layers 2 through 9: sequential feedforward model.
        out = self.seq_model(out)

        return out

    def fit(self, X, y):
        self.train()

        # TODO: set a random seed to invoke determinism.
        # cf. https://github.com/pytorch/pytorch/issues/11278

        X = torch.tensor(X, dtype=torch.float32).cuda()
        y = torch.tensor(y, dtype=torch.float32).cuda()

        # The build of PyTorch on Kaggle has a blog that prevents us from using
        # CyclicLR with ADAM. Cf. GH#19003.
        # optimizer = torch.optim.Adam(model.parameters(), lr=max_lr)
        # scheduler = torch.optim.lr_scheduler.CyclicLR(
        #     optimizer, base_lr=base_lr, max_lr=max_lr,
        #     step_size_up=300, step_size_down=300,
        #     mode='exp_range', gamma=0.99994
        # )
        optimizer = torch.optim.Adam(model.parameters(), lr=(self.base_lr + self.max_lr) / 2)
        batches = torch.utils.data.DataLoader(
            torch.utils.data.TensorDataset(X, y),
            batch_size=self.batch_size, shuffle=True
        )

        for epoch in range(self.n_epochs):
            for i, (X_batch, y_batch) in enumerate(batches):
                y_pred = model(X_batch).squeeze()
                # scheduler.batch_step()  # Disabled due to a bug, see above.
                loss = self.loss_fn(y_pred, y_batch)
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
            print(
                f"Epoch {epoch + 1}/{self.n_epochs}, Loss {loss.detach().numpy()}"
            )

    def predict(self, X):
        self.eval()
        with torch.no_grad():
            y_pred = model(torch.tensor(X, dtype=torch.float32).cuda())
        return y_pred.squeeze()

    def score(self, X, y):
        y_pred = self.predict(X)
        y = torch.tensor(y, dtype=torch.float32).cuda()
        return self.score_fn(y, y_pred)

model = FeedforwardTabularModel()
model.cuda()
model.fit(X_train_sample.values, y_train_sample.values)

This type of error typically occurs when there is a tensor in the model that should be on GPU but is on CPU instead. But as far as I can tell, I've already placed .cuda() calls at all of the necessary places: every time a torch.tensor is declared, and running model.cuda() before model.fit.

What is causing this error?

Output:
Someone on a separate forum provided the solution:

  Pytorch requires your to do self.module_name = module for things to work correctly. It's okay to keep them in a list. Just do something like setattr(self, 'emb_{}'.format(i), emb) for each step in that loop.

Because I was managing my embedding layers in a list, whereas PyTorch requires all layers be registered as an attribute on the model object, they were not automatically moved over to GPU memory when model.cuda() was called. Tricky!


Instruction: Multi dimensional inputs in pytorch Linear method?

Input:
When building a simple perceptron neural network we usuall passes a 2D matrix of input of format (batch_size,features) to a 2D weight matrix, similar to this simple neural network in numpy. I always assumed a Perceptron/Dense/Linear layer of a neural network only accepts an input of 2D format and outputs another 2D output. But recently I came across this pytorch model in which a Linear layer accepts a 3D input tensor and output another 3D tensor (o1 = self.a1(x)).

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.a1 = nn.Linear(4,4)
        self.a2 = nn.Linear(4,4)
        self.a3 = nn.Linear(9,1)
    def forward(self,x):
        o1 = self.a1(x)
        o2 = self.a2(x).transpose(1,2)
        output = torch.bmm(o1,o2)
        output = output.view(len(x),9)
        output = self.a3(output)
        return output

x = torch.randn(10,3,4)
y = torch.ones(10,1)

net = Net()

criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters())

for i in range(10):
    net.zero_grad()
    output = net(x)
    loss = criterion(output,y)
    loss.backward()
    optimizer.step()
    print(loss.item())

These are the question I have,

Is the above neural network a valid one? that is whether the model will train correctly?
Even after passing a 3D input x = torch.randn(10,3,4), why is the pytorch nn.Linear doesn't shows any error and gives a 3D output?

Output:
Newer versions of PyTorch allows nn.Linear to accept N-D input tensor, the only constraint is that the last dimension of the input tensor will equal in_features of the linear layer. The linear transformation is then applied on the last dimension of the tensor.
For instance, if in_features=5 and out_features=10 and the input tensor x has dimensions 2-3-5, then the output tensor will have dimensions 2-3-10.


Instruction: How to solve size mismatch of Multi Head Attention in pytorch?

Input:
Learning how to coding  Multi Head Attention in pytorch now,

I can't solve the problem of size_mismatch in case dimension of input tensor  have 4 dims.

I refer to the  def and class codes in
http://nlp.seas.harvard.edu/2018/04/03/attention.html

Sorry for the inconvenience,Can you give me advice?

#attention def and class

def clones(module, N):
    "Produce N identical layers."
    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])

def attention(query, key, value, mask=None, dropout=None):
    "Compute 'Scaled Dot Product Attention'"
    d_k = query.size(-1)
    scores = torch.matmul(query, key.transpose(-2, -1)) \
             / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    p_attn = F.softmax(scores, dim = -1)
    if dropout is not None:
        p_attn = dropout(p_attn)

    return torch.matmul(p_attn, value), p_attn

# MultiHead Attention class

class MultiHeadedAttention(nn.Module):
    def __init__(self, h, d_model, dropout=0.1):
        "Take in model size and number of heads."
        super(MultiHeadedAttention, self).__init__()
        assert d_model % h == 0
        # We assume d_v always equals d_k
        self.d_k = d_model // h
        self.h = h
        self.linears = clones(nn.Linear(d_model, d_model), 4)
        self.attn = None
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, query, key, value, mask=None):
        "Implements Figure 2"
        if mask is not None:
            # Same mask applied to all h heads.
            mask = mask.unsqueeze(1)
        nbatches = query.size(0)

        # 1) Do all the linear projections in batch from d_model => h x d_k
        query, key, value = \
            [l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)
             for l, x in zip(self.linears, (query, key, value))]

        # 2) Apply attention on all the projected vectors in batch.
        x, self.attn = attention(query, key, value, mask=mask,
                                 dropout=self.dropout)

        # 3) "Concat" using a view and apply a final linear.
        x = x.transpose(1, 2).contiguous() \
             .view(nbatches, -1, self.h * self.d_k)
        return self.linears[-1](x)

# create test_4_dim tensor
X=torch.randn(10,5,64,64)
X=X.view(X.shape[0],X.shape[1],X.shape[2]*X.shape[3])
#X:torch.Size([10, 5, 4096])
query_=X.transpose(2,1)
key_=X
value_=X

print("query:",query_.size())
print("key:",key_.size())
print("value:",value_.size())
#query: torch.Size([10, 4096, 5])
#key: torch.Size([10, 5, 4096])
#value: torch.Size([10, 5, 4096])

multihead_testmodel= MultiHeadedAttention(h=4,d_model=4096,dropout=0.1)
#print(multihead_model)

output=multihead_testmodel(query=query_,key=key_,value=value_)
print("model output:",output.size())

#size mismatch, m1: [40960 x 5], m2: [4096 x 4096] at #../aten/src/TH/generic/THTensorMath.cpp:197

in case tensor seze: torch.randn(5,64,64),this code has no error.

X=torch.randn(5,64,64)
#X=X.view(X.shape[0],X.shape[1],X.shape[2]*X.shape[3])

query_=X.transpose(2,1)
key_=X
value_=X

print("query:",query_.size())
print("key:",key_.size())
print("value:",value_.size())

#query: torch.Size([5, 64, 64])
#key: torch.Size([5, 64, 64])
#value: torch.Size([5, 64, 64])

multihead_model= MultiHeadedAttention(h=4,d_model=64,dropout=0.1)
temp_output=multihead_model(query=query_,key=key_,value=value_)
print(temp_output.size())
#torch.Size([5, 64, 64])

Output:
Looks like the code expects to get the same dimensions for query, key, and value, so if you don't transpose it fixes the issue:

query_ = X
key_ = X
value_ = X

You're right that there needs to be a transpose for the attention to work, but the code already handles this by calling key.transpose(-2, -1) in the attention implementation.


Instruction: What is the meaning of in-place in dropout

Input:
def dropout(input, p=0.5, training=True, inplace=False)

  inplace: If set to True, will do this operation in-place.

I would like to ask what is the meaning of in-place in dropout. What does it do?
Any performance changes when performing these operation?

Thanks

Output:
Keeping inplace=True will itself drop few values in the tensor input itself, whereas if you keep inplace=False, you will to save the result of droput(input) in some other variable to be retrieved.

Example:

import torch
import torch.nn as nn
inp = torch.tensor([1.0, 2.0, 3, 4, 5])

outplace_dropout = nn.Dropout(p=0.4)
print(inp)
output = outplace_dropout(inp)
print(output)
print(inp) # Notice that the input doesn't get changed here

inplace_droput = nn.Dropout(p=0.4, inplace=True)
inplace_droput(inp)
print(inp) # Notice that the input is changed now

PS: This is not related to what you have asked but try not using input as a variable name since input is a Python keyword. I am aware that Pytorch docs also does that, and it is kinda funny.


Instruction: PyTorch Confusion Matrix Plot

Input:
I have met a problem for plotting a confusion matrix. The upper and lower lines are put incorrectly.
When I plot it, it looks like this.

I think there is nothing wrong with my code, since I took it from this YouTube exactly.

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

# Graphical analytics
cm = confusion_matrix(train_set.targets, train_preds.argmax(dim=1))
names = ('T-shirt/top','Trouser','Pullover','Dress','Coat','Sandal','Shirt','Sneaker','Bag','Ankle boot')
plt.figure(figsize=(10,10))
plot_confusion_matrix(cm, names)

Output:
You can change manually the range of the y axis.

plt.ylim(-0.5, len(names) - 0.5)

For some reason heuristic for estimating the axis range does not get that you are not only interested in the points you are plotting but also in the (-0.5; +0.5) surrounding on both axes.

The lowest points have y coordinate 0, the top-most points have y coordinate len(names) - 1.


Instruction: How can I replace the entries of a tensor with the appropriate ranking of each entry?

Input:
suppose that I have the following tensor:

>> i = 3
>> j = 5
>> k = 2
>> sor = torch.randn(i,j,k)

>> sor
Out[20]:
tensor([[[ 0.5604, -0.9675],
         [-1.0953, -0.5615],
         [ 0.4250, -0.9176],
         [-1.6188, -1.0217],
         [-0.0778,  1.9407]],

        [[-0.1034, -0.7925],
         [-0.2955,  0.8058],
         [-0.5349,  1.1040],
         [ 1.1240,  0.8249],
         [ 0.0827, -1.2471]],

        [[ 0.5924,  0.4777],
         [-2.4640, -1.9527],
         [-0.4519,  0.4788],
         [-0.2308, -0.2368],
         [-1.6786,  0.1360]]])

suppose that for every fixed i and j, I want to compute the numeric rank of elements across k, and replace the elements of the tensor sor with those ranks. For instance, from the example above, I want to change the entry [ 0.5604, -0.9675], which is sor[0,0,:] , into [1, 2], since 0.5604 > -0.9675

Thank you,

Output:
I think you are looking for torch.argsort:

torch.argsort(sor, dim=2)

Out[ ]:
tensor([[[1, 0],
         [0, 1],
         [1, 0],
         [0, 1],
         [0, 1]],

        [[1, 0],
         [0, 1],
         [0, 1],
         [1, 0],
         [1, 0]],

        [[1, 0],
         [0, 1],
         [0, 1],
         [1, 0],
         [0, 1]]])


Instruction: pytorch question regarding backward argument used in blitz tutorial

Input:
A pytorch question, regarding backward(). In the pytorch blitz tutorial copied and pasted below, they pass in a vector [0.1, 1.0, 0.0001] to backward() . I can intuitively guess why vector [0.1, 1.0, 0.0001] shape passed in is [3] , but I do not understand where the values 0.1, 1.0, 0.0001 come from. Another tutorial I looked at passes in one such that backwards on a vector is done like this : L.backward(torch.ones(L.shape))

# copied from blitz tutorial
Now in this case y is no longer a scalar. torch.autograd could not compute the full Jacobian directly, but if we just want the vector-Jacobian product, simply pass the vector to backward as argument:

v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(v)

print(x.grad)

If anyone can explain the reasoning for [0.1, 1.0, 0.0001], I would appreciate it.

Output:
As the document says, implicitly, grad cannot be created for non-scalar outputs. y is non-scalar tensor, and you cannot y.backward() directly. But you can pass in a vector to backward to get the vector-Jacobian product. If you don't want to change the grads, you can pass in a vector with all elements are ones.

x = torch.tensor([2.,3.,4.], requires_grad=True)
y = x**2

y.backward() # error

y.backward(torch.tensor([1.,1.,1.])) # work
x.grad # tensor([4.,6.,8.])

# y.backward(torch.tensor([2.,2.,2.])) # change the passed vector.
# x.grad # tensor([8.,12.,16])


Instruction: Optimizer parameters missing in Pytorch

Input:
I have 2 nets sharing one optimizer using different learning rate. Simple code shown as below:

optim = torch.optim.Adam([
{'params': A.parameters(), 'lr': args.A},
{'params': B.parameters(), 'lr': args.B}])

Is this right? I ask this because when I check parameters in optimizer (using code below), I found only 2 parameters.

for p in optim.param_groups:
outputs = ''
for k, v in p.items():
    if k is 'params':
        outputs += (k + ': ' + str(v[0].shape).ljust(30) + ' ')
    else:
        outputs += (k + ': ' + str(v).ljust(10) + ' ')
print(outputs)

Only 2 parameters are printed:

params: torch.Size([16, 1, 80])        lr: 1e-05      betas: (0.9, 0.999) eps: 1e-08      weight_decay: 0          amsgrad: False

params: torch.Size([30, 10])           lr: 1e-05      betas: (0.9, 0.999) eps: 1e-08      weight_decay: 0          amsgrad: False

Actually, 2 nets have more than 100 parameters. I thought all parameters will be printed. Why is this happening? Thank you!

Output:
You only print the first tensor of each param groups:

if k is 'params':
    outputs += (k + ': ' + str(v[0].shape).ljust(30) + ' ')  # only v[0] is printed!

Try and print all the parameters:

if k is 'params':
    outputs += (k + ': ')
    for vp in v:
        outputs += (str(vp.shape).ljust(30) + ' ')


Instruction: How can I vectorize these nested loops in Python?

Input:
I am trying to vectorize the following:

n = torch.zeros_like(x)
for i in range(x.shape[0]):
    for j in range(x.shape[1]):
        for k in range(x.shape[2]):
            n[i, j, k] = p[i, x[i, j, k], j, k]

I tried doing something like

n = p[:, x, ...]

but I just get an error that I ran out of memory, which isn't very helpful. I think the problem with this is that instead of getting the value of x at the correct index it is trying to index the entirety of x, but I am not sure how I would go about fixing that if that is the problem.

Output:
This looks like a perfect use-case for broadcasted fancy indices. np.ogrid is a valuable tool here, or you can manually reshape your ranges:

i, j, k = np.ogrid[:x.shape[0], :x.shape[1], :x.shape[2]]
n = p[i, x, j, k]

This black magic works because the index into ogrid returns three arrays that broadcast into the same shape as x. Therefore the final extraction from p will have that shape. The indexing is trivial after that. Another way to write it is:

i = np.arange(x.shape[0]).reshape(-1, 1, 1)
j = np.arange(x.shape[1]).reshape(1, -1, 1)
k = np.arange(x.shape[2]).reshape(1, 1, -1)
n = p[i, x, j, k]


Instruction: how to add list of arrays (tensors)

Input:
I am defining a simple conv2d function to calculate the cross-correlation between input and kernel (both 2D tensor) as below:

import torch

def conv2D(X, K):
    h = K.shape[0]
    w = K.shape[1]
    ĥ = X.shape[0] - h + 1
    ŵ = X.shape[1] - w + 1
    Y = torch.zeros((ĥ, ŵ))
    for i in range (ĥ):
        for j in range (ŵ):
            Y[i, j] = (X[i: i+h, j: j+w]*K).sum()

    return Y

When X and K are of rank-3 tensor, I calculate the conv2d for each channel and then add them together as below:

def conv2D_multiple(X, K):
    cross = []
    result = 0
    for x, k in zip(X, K):
        cross.append(conv2D(x,k))

    for t in cross:
        result += t

    return result

To test my function:

X_2 = torch.tensor([[[0, 1, 2], [3, 4, 5], [6, 7, 8]],
                    [[1, 2, 3], [4, 5, 6], [7, 8, 9]]], dtype=torch.float32)
K_2 = torch.tensor([[[0, 1], [2, 3]], [[1, 2], [3, 4]]], dtype=torch.float32)

conv2D_multiple(X_2, K_2)

The results is:

tensor([[ 56.,  72.],
        [104., 120.]])

The result is as expected, however, I believe my second
for loop inside conv2D_multiple(X, K) function is redundant.  My question is how to sum (element wise)
tensors (arrays) in the list so I omit the second for loop.

Output:
Since your conv2D operates on a per slice behaviour, what you can do is allocate a 3D tensor so that when you use the first for loop, you store the results by taking each result and populating each slice.  You can then sum along the dimension of the slices using PyTorch's built-in torch.sum operator on the tensor to get the same result.  To make it palatable, I'll make the slice dimension dim=0.  Therefore, replace cross from being an initial empty list to a Torch tensor that is 3D to allow you to store the intermediate results, then compress along the slice dimension by summing.  We can get away with doing this as your initial implementation stored the intermediate results as a list of 2D tensors.  To make it easier, go to 3D and allow PyTorch to sum along the slice axis.

This will require that you define the correct dimensions for this 3D tensor first prior to looping:

def conv2D_multiple(X, K):
    h = K.shape[1]
    w = K.shape[2]
    ĥ = X.shape[1] - h + 1
    ŵ = X.shape[2] - w + 1
    c = X.shape[0]
    cross = torch.zeros((c, ĥ, ŵ), dtype=torch.float32)
    for i, (x, k) in enumerate(zip(X, K)):
        cross[i] = conv2D(x,k)

    result = cross.sum(dim=0)
    return result

Notice that for each slice you're iterating over between the input and kernel, instead of appending to a new list we directly place this into a slice in the intermediate tensor.  Once you store these results, sum along the slice axis to finally compress it into what you expect.  Running the new function above with your example inputs generates the same result.

If this isn't a desired result for you, another way is to simply take the list of tensors you created, build the intermediate tensor out of that by stacking them all together using torch.stack and sum.  By default it stacks along the first axis (dim=0):

def conv2D_multiple(X, K):
    cross = []
    result = 0
    for x, k in zip(X, K):
        cross.append(conv2D(x,k))

    cross = torch.stack(cross)
    result = cross.sum(dim=0)
    return result


Instruction: How to share the common parts of two models in pytorch？

Input:
I had a problem implementing the model with pytorch.

I want to build two models, some of which are shared, and share the encoder part like this

Model1: input_1 -> encoder -> decoder_1 -> ouput_1
Model2: input_2 -> encoder -> decoder_2 -> ouput_2

What I want to do is make the two models use the encoder part together, but the decoder part is not the same. I looked up about parameter sharing, but it seems to be somewhat different from the requirements here.

My own idea is to build a model that includes encode, decoder_1, decoder_2 and then choose which decoder to use based on input.

I'm not sure about this method, if possible, can you give simple examples for using the common parts of two models?

Output:
You could do something like:

import torch.nn as nn

class SharedModel(nn.Module):
    def __init__(self, mode):
        super(SharedModel, self).__init__()
        self.mode = mode # use 1 or 2

        self.encoder = ...
        self.decoder_1 = ...
        self.decoder_2 = ...

    def forward(self, x):
        x = self.encoder(x)
        if self.mode == 1:
            x = self.decoder_1(x)
        elif self.mode == 2:
            x = self.decoder_2(x)
        else:
            raise ValueError("Unkown mode.")
        return x


Instruction: Change the image size and range

Input:
I want to ask for data transforms if I have an image of size 28 * 28 and I want to resize it to be 32 *32, I know that this could be done with transforms.Resize() but I'm sure how.
Also for the normalization, if I want it to be within the range of [-1,1], I did it previously if I want it to be within [0,1] using transforms.Normalize((0.485,0.456,0.406),(0.229,0.224,0.225))

Output:
Don't rage, it's gonna be fine. Resizing MNIST to 32x32 height x width can be done like so:

import tempfile

import torchvision

dataset = torchvision.datasets.MNIST(
    root=tempfile.gettempdir(),
    download=True,
    train=True,
    # Simply put the size you want in Resize (can be tuple for height, width)
    transform=torchvision.transforms.Compose(
        [torchvision.transforms.Resize(32), torchvision.transforms.ToTensor()]
    ),
)

print(dataset[0][0].shape) # 1, 32, 32 (channels, width, height)

When it comes to normalization, you can see PyTorch's per-channel normalization source here. It depends whether you want it per-channel or in another form, but something along those lines should work (see wikipedia for formula of the normalization, here it's applied per-channel):

import dataclasses

@dataclasses.dataclass
class Normalize:
    maximum: typing.Tuple
    minimum: typing.Tuple
    low: int = -1
    high: int = 1

    def __call__(self, tensor):
        maximum = torch.as_tensor(self.maximum, dtype=dtype, device=tensor.device)
        minimum = torch.as_tensor(self.minimum, dtype=dtype, device=tensor.device)
        return self.low + (
            (tensor - minimum[:, None, None]) * (self.high - self.low)
        ) / (maximum[:, None, None] - minimum[:, None, None])

You would have to provide Tuple of minimum values and Tuple of maximum values (one value per channel for both) just like for standard PyTorch's torchvision normalization though. You could calculate those from data, for MNIST you could calculate them like this:

def per_channel_op(data, op=torch.max):
    per_sample, _ = op(data, axis=0)
    per_width, _ = op(per_sample, axis=1)
    per_height, _ = op(per_width, axis=1)
    return per_height

# Unsqueeze to add superficial channel for MNIST
# Divide cause they are uint8 type by default
data = dataset.data.unsqueeze(1).float() / 255

# Maximum over samples
maximum = per_channel_op(data) # value per channel, here
minimum = per_channel_op(data, op=torch.min) # only one value cause MNIST

And finally, to apply normalization on MNIST (watch out, as those will only have -1, 1 values as all pixels are black and white, will act differently on datasets like CIFAR etc.):

dataset = torchvision.datasets.MNIST(
    root=tempfile.gettempdir(),
    download=True,
    train=True,
    # Simply put the size you want in Resize (can be tuple for height, width)
    transform=torchvision.transforms.Compose(
        [
            torchvision.transforms.Resize(32),
            torchvision.transforms.ToTensor(),
            # Apply with Lambda your custom transformation
            torchvision.transforms.Lambda(Normalize((maximum,), (minimum,))),
        ]
    ),
)


Instruction: how to get jacobian with pytorch for log probability of multivariate normal distribution

Input:
I've drawn samples from a multivariate normal distribution and would like to get the gradient of their log probability with respect to the mean. Since there are many samples, this requires a Jacobian:

import torch

mu = torch.ones((2,), requires_grad=True)
sigma = torch.eye(2)
dist = torch.distributions.multivariate_normal.MultivariateNormal(mu, sigma)

num_samples=10
samples = dist.sample((num_samples,))
logprobs = dist.log_prob(samples)

Now I would like to get the derivative of each entry in logprobs with respect to each entry in mu.

A simple solution is a python loop:

grads = []
for logprob in logprobs:
    grad = torch.autograd.grad(logprob, mu, retain_graph=True)
    grads.append(grad)

If you stack the grads, the result is the desired Jacobian. Is there also built-in and vectorized support for this?

Related questions/internet ressources:

This is a huge topic, there are lots of related posts. Nevertheless, I think that this specific question (regarding distributions) hasn't been answered yet:

This question is basically the same as mine (but without example code and solution attempt), sadly it is unanswered: Pytorch custom function jacobian gradient
This question shows the calculation of a jacobian in pytorch, but I don't think the solution is applicable to my problem: Pytorch most efficient Jacobian/Hessian calculation It requires stacking the input in a way that seems incompatible with the distribution. I couldn't make it work.
This gist has some code snippets for Jacobians. In principle they are similar to the approach from the question above.

Output:
PyTorch 1.5.1 introduced a torch.autograd.functional.jacobian function. This computes the Jacobians of a function w.r.t. the input tensors. Since jacobian requires a python function as the first argument, using it requires some code restructuring.
import torch

torch.manual_seed(0)    # for repeatable results

mu = torch.ones((2,), requires_grad=True)
sigma = torch.eye(2)
num_samples = 10

def f(mu):
    dist = torch.distributions.multivariate_normal.MultivariateNormal(mu, sigma)
    samples = dist.sample((num_samples,))
    logprobs = dist.log_prob(samples)
    return logprobs

grads = torch.autograd.functional.jacobian(f, mu)

print(grads)

tensor([[-1.1258, -1.1524],
        [-0.2506, -0.4339],
        [ 0.5988, -1.5551],
        [-0.3414,  1.8530],
        [ 0.4681, -0.1577],
        [ 1.4437,  0.2660],
        [ 1.3894,  1.5863],
        [ 0.9463, -0.8437],
        [ 0.9318,  1.2590],
        [ 2.0050,  0.0537]])


Instruction: PyTorch DataLoader returns the batch as a list with the batch as the only entry. How is the best way to get a tensor from my DataLoader

Input:
I currently have the following situation where I want to use DataLoader to batch a numpy array:

import numpy as np
import torch
import torch.utils.data as data_utils

# Create toy data
x = np.linspace(start=1, stop=10, num=10)
x = np.array([np.random.normal(size=len(x)) for i in range(100)])
print(x.shape)
# >> (100,10)

# Create DataLoader
input_as_tensor = torch.from_numpy(x).float()
dataset = data_utils.TensorDataset(input_as_tensor)
dataloader = data_utils.DataLoader(dataset,
                                   batch_size=100,
                                  )
batch = next(iter(dataloader))

print(type(batch))
# >> <class 'list'>

print(len(batch))
# >> 1

print(type(batch[0]))
# >> class 'torch.Tensor'>

I expect the batchto be already a torch.Tensor. As of now I index the batch like so, batch[0] to get a Tensor but I feel this is not really pretty and makes the code harder to read.

I found that the DataLoader takes a batch processing function called collate_fn.  However, setting data_utils.DataLoader(..., collage_fn=lambda batch: batch[0]) only changes the list to a tuple (tensor([ 0.8454, ..., -0.5863]),) where the only entry is the batch as a Tensor.

You would help me a lot by helping me finding out how to elegantly transform the batch to a tensor (even if this would include telling me that indexing the single entry in batch is okay).

Output:
Sorry for inconvenience with my answer.

Actually, you don't have to create Dataset from your tensor, you can pass torch.Tensor directly as it implements __getitem__ and __len__, so this is sufficient:

import numpy as np
import torch
import torch.utils.data as data_utils

# Create toy data
x = np.linspace(start=1, stop=10, num=10)
x = np.array([np.random.normal(size=len(x)) for i in range(100)])

# Create DataLoader
dataset = torch.from_numpy(x).float()
dataloader = data_utils.DataLoader(dataset, batch_size=100)
batch = next(iter(dataloader))


Instruction: KeyError when enumerating over dataloader

Input:
I'm trying to iterate over a pytorch dataloader initialized as follows:

trainDL = torch.utils.data.DataLoader(X_train,batch_size=BATCH_SIZE, shuffle=True, **kwargs)

where X_train is a pandas dataframe like this one:

So, I'm not being able to do the following statement, since I'm getting a KeyError in the 'enumerate':

for batch_idx, (data, _) in enumerate(trainDL):
    {stuff}

has anyone a clue of what's happening?

EDIT:

The error I get is:

KeyError                                  Traceback (most recent call last)
~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2896             try:
-> 2897                 return self._engine.get_loc(key)
   2898             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 40592

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-63-95142e0748bb> in <module>
----> 1 for batch_idx, (data, _) in enumerate(trainDL):
      2     print(".")

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    344     def __next__(self):
    345         index = self._next_index()  # may raise StopIteration
--> 346         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    347         if self._pin_memory:
    348             data = _utils.pin_memory.pin_memory(data)

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/pandas/core/frame.py in __getitem__(self, key)
   2993             if self.columns.nlevels > 1:
   2994                 return self._getitem_multilevel(key)
-> 2995             indexer = self.columns.get_loc(key)
   2996             if is_integer(indexer):
   2997                 indexer = [indexer]

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2897                 return self._engine.get_loc(key)
   2898             except KeyError:
-> 2899                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2900         indexer = self.get_indexer([key], method=method, tolerance=tolerance)
   2901         if indexer.ndim > 1 or indexer.size > 1:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 40592

Output:
You have to create torch.utils.data.Dataset wrapping your dataset.
For example:
from torch.utils.data import Dataset

class PandasDataset(Dataset):
    def __init__(self, dataframe):
        self.dataframe = dataframe

    def __len__(self):
        return len(self.dataframe)

    def __getitem__(self, index):
        return self.dataframe.iloc[index]

Pass this object to DataLoader instantiated by your pandas dataframe and you should be fine.
Example usage with DataLoader:
import pandas as pd

df = pd.read_csv("data.csv")
dataset = PandasDataset(df)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=16)
for sample in dataloader:
    ...


Instruction: PyTorch C++ Extensions: Accessing data for Half Tensors

Input:
I'm trying to write a C++/CUDA extension for PyTorch using the C++ Tensor API, and I would like my code to work with both float32 and float16 (half precision). I'm not sure how to access the data pointer for half tensors coming from Python.

Here's how I do it for float tensors:

// Access data pointer for float Tensor A
torch::Tensor A;
float* ptr = A.data<float>();

Here's what I've tried for half tensors:

// CUDA float 16 type
// undefined symbol: _ZNK2at6Tensor4dataI6__halfEEPT_v
A.data<__half>();

// PyTorch float16 type
// error: no instance of function template "at::Tensor::data"
A.data<torch::ScalarType::Half>();

// Casting to __half*
// This compiles but throws and error if the requested pointer type doesn't match the Tensor type:
// RuntimeError: expected scalar type Float but found Half
(__half*)(A.data<float>());

I tried looking into the C++ api source code, but couldn't find anything else that looks like a float16 type.

System information:
Python 3.6.2
PyTorch 1.0.1

Output: The correct type turned out to be at::Half.


Instruction: How to make prediction from train Pytorch and PytorchText model?

Input:
General speaking, after I have successfully trained a text RNN model with Pytorch, using PytorchText to leverage data loading on an origin source, I would like to test with other data sets (a sort of blink test) that are from different sources but the same text format.

First I defined a class to handle the data loading.

class Dataset(object):
    def __init__(self, config):
        # init what I need

    def load_data(self, df: pd.DataFrame, *args):
        # implementation below
        # Data format like `(LABEL, TEXT)`

    def load_data_but_error(self, df: pd.DataFrame):
        # implementation below
        # Data format like `(TEXT)`

Here is the detail of load_data which I load data that trained successfully.

TEXT = data.Field(sequential=True, tokenize=tokenizer, lower=True, fix_length=self.config.max_sen_len)
LABEL = data.Field(sequential=False, use_vocab=False)

datafields = [(label_col, LABEL), (data_col, TEXT)]

# split my data to train/test
train_df, test_df = train_test_split(df, test_size=0.33, random_state=random_state)

train_examples = [data.Example.fromlist(i, datafields) for i in train_df.values.tolist()]
train_data = data.Dataset(train_examples, datafields)

# split train to train/val
train_data, val_data = train_data.split(split_ratio=0.8)

# build vocab
TEXT.build_vocab(train_data, vectors=Vectors(w2v_file))
self.word_embeddings = TEXT.vocab.vectors
self.vocab = TEXT.vocab

test_examples = [data.Example.fromlist(i, datafields) for i in test_df.values.tolist()]
test_data = data.Dataset(test_examples, datafields)

self.train_iterator = data.BucketIterator(
            (train_data),
            batch_size=self.config.batch_size,
            sort_key=lambda x: len(x.title),
            repeat=False,
            shuffle=True)

self.val_iterator, self.test_iterator = data.BucketIterator.splits(
    (val_data, test_data),
    batch_size=self.config.batch_size,
    sort_key=lambda x: len(x.title),
    repeat=False,
    shuffle=False)

Next is my code (load_data_but_error) to load others source but causing error

TEXT = data.Field(sequential=True, tokenize=tokenizer, lower=True, fix_length=self.config.max_sen_len)
datafields = [('title', TEXT)]

examples = [data.Example.fromlist(i, datafields) for i in df.values.tolist()]
blink_test = data.Dataset(examples, datafields)

self.blink_test = data.BucketIterator(
    (blink_test),
    batch_size=self.config.batch_size,
    sort_key=lambda x: len(x.title),
    repeat=False,
    shuffle=True)

When I was executing code, I had an error AttributeError: 'Field' object has no attribute 'vocab' which has a question at here but it doesn't like my situation as here I had vocab from load_data and I want to use it for blink tests.

My question is what the correct way to load and feed new data with a trained PyTorch model for testing current model is?

Output:
What I need are

to keep TEXT in load_data and reuse in load_data_but_error by assigning to class variables
add train=True to object data.BucketIterator on load_data_but_error function


Instruction: Custom distance loss function in Pytorch?

Input:
I want to implement the following distance loss function in pytorch. I was following this https://discuss.pytorch.org/t/custom-loss-functions/29387/4 thread from the pytorch forum

np.linalg.norm(output - target)
# where output.shape = [1, 2] and target.shape = [1, 2]

So I have implemented the loss function like this

def my_loss(output, target):
    loss = torch.tensor(np.linalg.norm(output.detach().numpy() - target.detach().numpy()))
    return loss

with this loss function, calling backwards gives runtime error

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

My entire code looks like this

model = nn.Linear(2, 2)

x = torch.randn(1, 2)
target = torch.randn(1, 2)
output = model(x)

loss = my_loss(output, target)
loss.backward()   <----- Error here

print(model.weight.grad)

PS: I am aware of the pairwise loss of pytorch but due to some limitation of it, I have to implement it myself.

Following the pytorch source code I have tried the following,

class my_function(torch.nn.Module): # forgot to define backward()
    def forward(self, output, target):

        loss = torch.tensor(np.linalg.norm(output.detach().numpy() - target.detach().numpy()))
        return loss

model = nn.Linear(2, 2)
x = torch.randn(1, 2)
target = torch.randn(1, 2)
output = model(x)

criterion = my_function()

loss = criterion(output, target)

loss.backward()
print(model.weight.grad)

And I get the Run time error

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

How can I implement the loss function correctly?

Output:
This happens because, in the loss function, you are detaching tensors. You had to detach because you wanted to use np.linalg.norm. This breaks the graph and you get the error that tensors don't have grad fn.

You can replace

loss = torch.tensor(np.linalg.norm(output.detach().numpy() - target.detach().numpy()))

by torch operations as

loss = torch.norm(output-target)

This should work fine.


Instruction: PyTorch Softmax Output Doesn't Sum to 1

Input:
Cross posting my question from the PyTorch forum:

I started receiving negative KL divergences between a target Dirichlet distribution and my model’s output Dirichlet distribution. Someone online suggested that this might be indicative that the parameters of the Dirichlet distribution don’t sum to 1. I thought this was ridiculous since the output of the model is passed through

output = F.softmax(self.weights(x), dim=1)

But after looking into it more closely, I found that torch.all(torch.sum(output, dim=1) == 1.) returns False! Looking at the problematic row, I see that it is tensor([0.0085, 0.9052, 0.0863], grad_fn=<SelectBackward>). But torch.sum(output[5]) == 1. produces tensor(False).

What am I misusing about softmax such that output probabilities do not sum to 1?

This is PyTorch version 1.2.0+cpu. Full model is copied below:

import torch
import torch.nn as nn
import torch.nn.functional as F

def assert_no_nan_no_inf(x):
    assert not torch.isnan(x).any()
    assert not torch.isinf(x).any()

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.weights = nn.Linear(
            in_features=2,
            out_features=3)

    def forward(self, x):
        output = F.softmax(self.weights(x), dim=1)
        assert torch.all(torch.sum(output, dim=1) == 1.)
        assert_no_nan_no_inf(x)
        return output

Output:
This is most probably due to the floating point numerical errors due to finite precision.

Instead of checking strict inequality you should check the mean square error or something to be within an acceptable limit.

For ex: I get torch.norm(output.sum(dim=1)-1)/N to be less than 1e-8. N is the batch size.


Instruction: Install pytorch from the source using pip

Input:
I am trying to install pytorch on a remote server. It has CentOS 6.5 and according to this link
it has stopped support for CentOS 6. So, I am trying to install it via source.
The recommended method to install is by anaconda but the thing is I am getting plenty of issues while installing anaconda as it is messing with the remote server path's alot so I have decided to use pip. But I have issues regarding converting some conda commands in pip which are as below-

conda install -c pytorch magma-cuda90
The above command is mentioned before the pytorch cloning step and it given me an error that Could not open requirements file: [Errno 2] No such file or directory: 'pytorch'
The other issue I am facing is below-
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
Wht should be the alternative for CMAKE_PREFIX_PATH in pip?

Output:
According to your Python version, you can try to install from the wheel files.

pip install https://download.pytorch.org/whl/cu101/torch-1.3.0-cp36-cp36m-manylinux1_x86_64.whl   --user # For torch
pip install https://download.pytorch.org/whl/cu101/torchvision-0.4.1-cp36-cp36m-linux_x86_64.whl  --user # For torchvision

If it fails you may want to check your glibc version:

ldd --version

because PyTorch is supported on Linux distributions that use glibc >= v2.17.

For your question:

  What should be the alternative for `CMAKE_PREFIX_PATH in pip ?

CMAKE_PREFIX_PATH act as a build directive to indicate where to find modules needed for the build. In your case (installation as non-root with the --user flag) it's probably:

~/.local/lib/python3.6/site-packages

You can verify the exact location with the following command:

python -c "import site; print(site.getsitepackages()[0])"

As a side note your compilation will more likely fail if you still don't have the minimum required version of glibc.


Instruction: How to fix this error: list indices must be integers or slices, not tuple

Input:
Suppose I have a list of tensors called outputs

>> outputs[2][0][0,:,:]
Out[20]:
tensor([[ 14.0448,  -5.1494,  -0.1780,  ...,  10.1937,  -8.9158,  -5.3964],
        [ 32.0382,  -0.5201,  29.9942,  ..., -18.8268, -23.1068,  23.9745],
        [-24.5911,  14.7233,  -6.3053,  ...,  -5.8131,  -3.3088,   0.5685],
        [  0.8842, -14.8318,   6.7204,  ...,  17.7127,   7.3332,   3.7249],
        [  2.0654, -16.5236,  38.3582,  ..., -23.1663,  -5.1202,  13.6506]],
       grad_fn=<SliceBackward>)

>> outputs[2][1][0,:,:]
Out[24]:
tensor([[-0.1260,  0.0463, -0.3362,  ...,  0.1089, -0.2454,  0.0140],
        [ 0.5050, -0.0750, -0.1639,  ..., -0.0020, -0.0521, -0.3224],
        [-0.5311,  0.4526,  0.0079,  ..., -0.0654, -0.1255, -0.0012],
        [ 0.0728, -0.1219,  0.0905,  ...,  0.1354,  0.2730, -0.1186],
        [-0.0680, -0.5570,  0.0295,  ..., -0.2411, -0.1690,  0.0331]],
       grad_fn=<SliceBackward>)

When I try to do:

>> outputs[2][0:1][0,:,:]

python generates an error, and the error message is

TypeError: list indices must be integers or slices, not tuple

How can I fix this error?

Thank you,

Output:
outputs[2][0] and outputs[2][1] both return an object (tensor I suppose).
outputs[2][0:1] returns a list of those objects.
What I think you are looking for is something like outputs[2][0:1][:,0,:,:] or [a[0,:,:] for a in outputs[2][0:1]]


Instruction: Using torch.nn.Embedding for GloVe: should we fine-tune the embeddings or just use them as they are?

Input:
while transfer learning / fine-tuning recent language models, such as BERT and XLNET, is by far a very common practice, how is this for GloVe?

Basically, I see two options when using GloVe to get dense vector representations that can be used by downstream NNs.

1) Fine-tune GloVe embeddings (in pytorch terms, gradient enabled)

2) Just use the embeddings without gradient.

For instance, given GloVe's embeddings matrix, I do

embed = nn.Embedding.from_pretrained(torch.tensor(embedding_matrix, dtype=torch.float))
...
dense = nn.Linear(...)

Is it best practice to solely use GloVe to get vector representation (and only train the dense layer and potentially other layers) or would one fine-tune the embeddings matrix, too?

Output:
You should absolutely fine-tune your word embedding matrix. Here is the thing, when you initialize the word embedding matrix with the GloVe word embeddings, your word embeddings will already capture most of the semantic properties of the data. However, you want your word embeddings to be tailored to the task your solving i.e task specific (Check Yang). Now, assuming that you don't have enough data in your dataset, you can't learn the word embedding matrix on your own (If you initialize the word embedding matrix with random vectors). Because of that, you want to initialize it with vectors that have been trained on huge datasets and are general.
One really important thing to keep in mind → Because the rest of your model is going to be initialized randomly, when you start training your word embedding matrix may suffer from catastrophic forgetting (Check the work of Howard and Ruder and Kirkpatrick et al.), i.e., the gradients will be huge because your model will drastically underfit the data for the first few batches, and you will lose the initial vectors completely. You can overcome this by:

For the first several epochs don't fine-tune the word embedding matrix, just keep it as it is: embeddings = nn.Embedding.from_pretrained(glove_vectors, freeze=True).

After the rest of the model has learned to fit your training data, decrease the learning rate, unfreeze the your embedding module embeddings.weight.requires_grad = True, and continue training.

By following the above mentioned steps, you will get the best of both worlds. In other words, your word embeddings will still capture semantic properties while being tailored for your own downstream task. Finally, there are works (Check Ye Zhang for example) showing that it is fine to fine-tune immediately, but I would opt for the safer option.