Instruction: Iterating over subsets from torch.utils.data.random_split

Input:
I am currently loading a folder with AI training data in it. The subfolders represent the label names with the corresponding images inside. This works well by using pyTorch's ImageFolder loader.

def load_dataset():
    data_path = 'C:/example_folder/'

    train_dataset_manual = torchvision.datasets.ImageFolder(
        root=data_path,
        transform=torchvision.transforms.ToTensor()
    )

    train_loader_manual = torch.utils.data.DataLoader(
        train_dataset_manual,
        batch_size=1,
        num_workers=0,
        shuffle=True
    )

    return train_loader_manual

full_dataset = load_dataset()

Now I want to have this dataset split into a training and a test data set. I am using the random_split function for this:

training_data_size = 0.8

train_size = int(training_data_size * len(full_dataset))
test_size = len(full_dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])

The full_dataset is an object of type torch.utils.data.dataloader.DataLoader. I can iterate through it with a loop like this:

for batch_idx, (data, target) in enumerate(full_dataset):
    print(batch_idx)

The train_dataset is an object of type torch.utils.data.dataset.Subset. If I try to loop through it, I get:

  TypeError 'DataLoader' object is not subscriptable:

for batch_idx, (data, target) in enumerate(train_dataset):
    print(batch_idx)

How can I loop through it? I am relatively new to Python.

Thanks!

Output:
You need to apply random_split to a Dataset not a DataLoader. The dataset used to define the DataLoader is available in the DataLoader.dataset member.

For example you could do

train_dataset, test_dataset = torch.utils.data.random_split(full_dataset.dataset, [train_size, test_size])

train_loader = DataLoader(train_dataset, batch_size=1, num_workers=0, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=1, num_workers=0, shuffle=False)

Then you can iterate over train_loader and test_loader as expected.


Instruction: PyTorch LSTM: RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 1219 and 440 in dimension 1

Input:
I have a basic PyTorch LSTM:
import torch.nn as nn
import torch.nn.functional as F

class BaselineLSTM(nn.Module):
    def __init__(self):
        super(BaselineLSTM, self).__init__()

        self.lstm = nn.LSTM(input_size=13, hidden_size=13)

    def forward(self, x):
        x = self.lstm(x)

        return x

For my data, I have:
train_set = CorruptedAudioDataset(corrupted_path, train_set=True)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=128, shuffle=True, **kwargs)

My CorruptedAudioDataset has:
    def __getitem__(self, index):
        corrupted_sound_file = SoundFile(self.file_paths[index])
        corrupted_samplerate = corrupted_sound_file.samplerate
        corrupted_signal_audio_array = corrupted_sound_file.read()

        clean_path = self.file_paths[index].split('/')
        # print(self.file_paths[index], clean_path)
        clean_sound_file = SoundFile(self.file_paths[index])
        clean_samplerate = clean_sound_file.samplerate
        clean_signal_audio_array = clean_sound_file.read()

        corrupted_mfcc = mfcc(corrupted_signal_audio_array, samplerate=corrupted_samplerate)
        clean_mfcc = mfcc(clean_signal_audio_array, samplerate=clean_samplerate)

        print('return', corrupted_mfcc.shape, clean_mfcc.shape)
        return corrupted_mfcc, clean_mfcc

My training loop looks like:
    model = BaselineLSTM()
    for epoch in range(300):
        for inputs, outputs in train_loader:
            print('inputs', inputs)

And that's the line that I get the error on:
  File "train_lstm_baseline.py", line 47, in train
    for inputs, outputs in train_loader:
...
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 1219 and 440 in dimension 1 at ../aten/src/TH/generic/THTensor.cpp:612

Output:
This exception is thrown basically because you're loading batches with different shapes. As they're stored in the same tensor, all samples must have the same shape. In this case, you have an input in dimension 0 with 1219 and 440, which is not possible. For example, you have something like:
torch.Size([1, 1219])
torch.Size([1, 440])
torch.Size([1, 550])
...

You must have:
torch.Size([1, n])
torch.Size([1, n])
torch.Size([1, n])
...

The easiest way to solve this problem is setting batch_size=1. However, it may delay your code.
The best way is setting the data to the same shape. In this case, you need to assess your problem to check if it's possible.


Instruction: How can I resize a PyTorch tensor with a sliding window?

Input: I have a tensor with size: torch.Size([118160, 1]). What I want to do is split it up into n tensors with 100 elements each, sliding by 50 elements at a time. What's the best way to achieve this with PyTorch?

Output:
A possible solution is:

window_size = 100
stride = 50
splits = [x[i:min(x.size(0),i+window_size)] for i in range(0,x.size(0),stride)]

However, the last few elements will be shorter than window_size. If this is undesired, you can do:

splits = [x[i:i+window_size] for i in range(0,x.size(0)-window_size+1,stride)]

EDIT:

A more readable solution:

# if keep_short_tails is set to True, the slices shorter than window_size at the end of the result will be kept
def window_split(x, window_size=100, stride=50, keep_short_tails=True):
  length = x.size(0)
  splits = []

  if keep_short_tails:
    for slice_start in range(0, length, stride):
      slice_end = min(length, slice_start + window_size)
      splits.append(x[slice_start:slice_end])
  else:
    for slice_start in range(0, length - window_size + 1, stride):
      slice_end = slice_start + window_size
      splits.append(x[slice_start:slice_end])

  return splits


Instruction: Transformers summarization with Python Pytorch - how to get longer output?

Input:
I use Ai-powered summarization from https://github.com/huggingface/transformers/tree/master/examples/summarization - state of the art results.

Should i train it myself to get summary output longer than used in original huggingface github training script?
:

python run_summarization.py \
    --documents_dir $DATA_PATH \
    --summaries_output_dir $SUMMARIES_PATH \ # optional
    --no_cuda false \
    --batch_size 4 \
    --min_length 50 \
    --max_length 200 \
    --beam_size 5 \
    --alpha 0.95 \
    --block_trigram true \
    --compute_rouge true

When i do inference with

--min_length 500 \
--max_length 600 \

I got a good output for 200 tokens, but the rest of the text is

. . . [unused7] [unused7] [unused7] [unused8] [unused4] [unused7] [unused7]  [unused4] [unused7] [unused8]. [unused4] [unused7] . [unused4] [unused8] [unused4] [unused8].  [unused4]  [unused4] [unused8]  [unused4] . .  [unused4] [unused6] [unused4] [unused7] [unused6] [unused4] [unused8] [unused5] [unused4] [unused7] [unused4] [unused4] [unused7]. [unused4] [unused6]. [unused4] [unused4] [unused4] [unused8]  [unused4] [unused7]  [unused4] [unused8] [unused6] [unused4]   [unused4] [unused4]. [unused4].  [unused5] [unused4] [unused8] [unused7] [unused4] [unused7] [unused9] [unused4] [unused7]  [unused4] [unused7] [unused5] [unused4]  [unused5] [unused4] [unused6]  [unused4]. .  . [unused5]. [unused4]  [unused4]   [unused4] [unused6] [unused5] [unused4] [unused4]  [unused6] [unused4] [unused6]  [unused4] [unused4] [unused5] [unused4]. [unused5]  [unused4] . [unused4]  [unused4] [unused8] [unused8] [unused4]  [unused7] [unused4] [unused8]  [unused4] [unused7]  [unused4] [unused8]  [unused4]  [unused8] [unused4] [unused6]

Output:
The short answer is: Yes, probably.

To explain this in a bit more detail, we have to look at the paper behind the implementation: In Table 1, you can clearly see that most of their generated headlines are much shorter than what you are trying to initialize. While that alone might not be an indicator that you couldn't generate anything longer, we can go even deeper and look at the meaning of the [unusedX] tokens, as described by BERT dev Jacob Devlin:

  Since [the [unusedX] tokens] were not used they are effectively randomly initialized.

Further, the summariazation paper describes

  Position embeddings in the original BERT model have a maximum length
  of 512; we over-come this limitation by adding more position
  em-beddings  that  are  initialized randomly and fine-tuned with other
  parameters in the encoder.

This is a strong indicator that past a certain length, they are likely falling back to the default initialization, which is unfortunately random. The question is whether you can still salvage the previous pre-training, and simply fine-tune to your objective, or whether it is better to just start from scratch.


Instruction: View on portion of tensor

Input:
I have a multiple dimensions tensor, let's take this simple one as example:

out = torch.Tensor(3, 4, 5)

I have to get a portion/subpart of this tensor out[:,0,:] and then apply the method view(-1), but it's not possible:

 out[:,0,:].view(-1)

RuntimeError: invalid argument 2: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Call .contiguous() before .view(). at ../aten/src/TH/generic/THTensor.cpp:203

A solution is to clone the subpart:

 out[:,0,:].clone().view(-1)

Is there a better/faster solution than cloning?

Output:
What you did will work fine. That said, a more portable approach would be to use reshape which will return a view when possible, but will create a contiguous copy if necessary. That way it will do the fastest thing possible. In your case the data must be copied, but by always using reshape there are cases where a copy won't be produced.

So you could use

out[:,0,:].reshape(-1)

Gotcha

There's one important gotcha here. If you perform in-place operations on the output of reshape then that may or may not affect the original tensor, depending on whether or not a view or copy was returned.

For example, assuming out is already contiguous then in this case

>>> x = out[:,0,:].reshape(-1)    # returns a copy
>>> x[0] = 10
>>> print(out[0,0,0].item() == 10)
False

x is a copy so changes to it don't affect out. But in this case

>>> x = out[:,:,0].reshape(-1)     # returns a view
>>> x[0] = 10
>>> print(out[0,0,0].item() == 10)
True

x is a view, so in-place changes to x will change out as well.

Alternatives

A couple alternative are

out[:,0,:].flatten()    # .flatten is just a special case of .reshape

and

out[:,0,:].contiguous().view(-1)

Though if you want the fastest approach I recommend against the latter method using contiguous().view since, in general, it is more likely than reshape or flatten to return a copy. This is because contiguous will create a copy even if the underlying data has the same number of bytes between subsequent entries. Therefore, there's a difference between

out[:,:,0].contiguous().view(-1)       # creates a copy

and

out[:,:,0].flatten()                   # creates a non-contiguous view (b/c underlying data has uniform spacing of out.shape[2] values between entries)

where the contiguous().view approach forces a copy since out[:,:,0] is not contiguous, but flatten/reshape would create a view since the underlying data is uniformly spaced.

Sometimes contiguous() won't create a copy, for example compare

out[0,:,:].contiguous().view(-1)       # creates a view b/c out[0,:,:] already is contiguous

and

out[0,:,:].flatten()                   # creates a view

which both produce a view of the original data without copying since out[0,:,:] is already contiguous.

If you want to ensure that the out is decoupled completely from its flattened counterpart then the original approach using .clone() is the way to go.


Instruction: How do I represent a PyTorch LSTM 3D Tensor?

Input:
As per the docs, I see that Pytorch’s LSTM expects all of its inputs to be 3D tensors. I am trying to do a simple sequence-to-sequence LSTM and I have:

class BaselineLSTM(nn.Module):
    def __init__(self):
        super(BaselineLSTM, self).__init__()

        self.lstm = nn.LSTM(input_size=100, hidden_size=100)

    def forward(self, x):
        print('x', x)
        x = self.lstm(x)

        return x

My x.size() is torch.Size([100, 1]). I expect I need a third dimension somehow, but I'm unsure as to what it actually means. Any help would be greatly appreciated.

Output:
The input shape is further elaborated on in this Pytorch docs, in the Inputs: input, (h_0, c_0)section. The first dimension of the input tensor is expected to correspond to the sequence length, the second dimension the batch size, and the third, the input size.

So for your example, the input tensor x should actually be of size (seq_length, batch_size, 100).

Here is a detailed thread on the Pytorch forum for more details: https://discuss.pytorch.org/t/why-3d-input-tensors-in-lstm/4455/9


Instruction: How do I calculate cross-entropy from probabilities in PyTorch?

Input:
By default, PyTorch's cross_entropy takes logits (the raw outputs from the model) as the input. I know that CrossEntropyLoss combines LogSoftmax (log(softmax(x))) and NLLLoss (negative log likelihood loss) in one single class. So, I think I can use NLLLoss to get cross-entropy loss from probabilities as follows:

  true labels: [1, 0, 1]
  probabilites: [0.1, 0.9], [0.9, 0.1], [0.2, 0.8]

where, y_i,j denotes the true value i.e. 1 if sample i belongs to class j and 0 otherwise.
and p_i,j denotes the probability predicted by your model of sample i belonging to class j.

If I calculate by hand, it turns out to be:

>>> -(math.log(0.9) + math.log(0.9) + math.log(0.8))
0.4338

Using PyTorch:

>>> labels = torch.tensor([1, 0, 1], dtype=torch.long)
>>> probs = torch.tensor([[0.1, 0.9], [0.9, 0.1], [0.2, 0.8]], dtype=torch.float)
>>> F.nll_loss(torch.log(probs), labels)
tensor(0.1446)

What am I doing wrong? Why is the answer different?

Output:
There is a reduction parameter for all loss functions in the PyTorch. As you can see from the documentation default reduction parameter is 'mean' which divides the sum with number of elements in the batch. To get  a summation behavior (0.4338) as you want, you should give the reduction parameter as following:

F.nll_loss(torch.log(probs), labels,reduction='sum')


Instruction: Does changing image channel break the image?

Input:
so I have images with the format(width,height,channel). My original channel is in rgb. So I Load the images in grayscale

for r, d, file in tqdm(os.walk(path)):
  for i in tqdm(file):
    if i[0:2]=="01":
      dist_one.append(cv2.imread(os.path.join(path,i),cv2.IMREAD_GRAYSCALE))
    else:
      dist_two.append(cv2.imread(os.path.join(path,i),cv2.IMREAD_GRAYSCALE))

Suppose the images have the shape (187, 187). So I add a channel using the code

g = np.expand_dims(dist_one[0], axis=0)

But this breaks the images when I try to plot the image.

  TypeError                                 Traceback (most recent call
  last)  in ()       1 import
  matplotlib.pyplot as plt       2  ----> 3
  plt.imshow(dist_one[0],cmap='gray')       4 plt.show()  5
  frames/usr/local/lib/python3.6/dist-packages/matplotlib/image.py in
  set_data(self, A)     688                 or self._A.ndim == 3 and
  self._A.shape[-1] in [3, 4]):     689             raise
  TypeError("Invalid shape {} for image data" --> 690
  .format(self._A.shape))     691      692 if self._A.ndim == 3:
  TypeError: Invalid shape (1, 187, 187) for image data

But it works when the channel is put last.
g = np.expand_dims(dist_one[0], axis=-1).
Whats the reason for this?
I need the channel at the first for pytorch. Or am I suppose to train the model with broken images?

Output: Pytorch requires [C, H, W], whereas numpy and matplotlib (which is where the error is being thrown) require the image to be [H, W, C]. To fix this issue, I'd suggest plotting g obtained by doing this g = np.expand_dims(dist_one[0], axis=-1). Before sending it to PyTorch, you can do one of two things. You can use 'g = torch.tensor(g).permute(2, 0, 1)` which will results in a [C, H, W] tensor or you can use the PyTorch Dataset and Dataloader setup, which handles other things like batching for you.


Instruction: Add Training and Testing Accuracy to a Simple Neural Network in PyTorch

Input:
I know this is a primitive question but what should I add in my code for it to output the training accuracy of the Neural Network in addition to the loss, I checked PyTorch tutorials and they show how to add training/testing accuracy in image classification but I do not know how to do that in my simple XOR solving NN, below is the code:

# Step 1: importing our dependencies
import torch
from torch.autograd import Variable
import numpy as np

# Our data
x = Variable(torch.Tensor([[0, 0, 1], [0, 1, 1], [1, 0, 1], [0, 1, 0], [1, 0, 0],
                           [1, 1, 1], [0, 0, 0]]))
y = Variable(torch.Tensor([[0], [1], [1], [1], [1], [0], [0]]))

# Step 2: building our class model
class NeuralNetwork(torch.nn.Module):
def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.linear_ij = torch.nn.Linear(3, 4)
    self.linear_jk = torch.nn.Linear(4, 1)

def forward(self, x):
    matmul = self.linear_ij(x)
    activation = torch.sigmoid(matmul)
    matmul = self.linear_jk(activation)
    prediction = torch.sigmoid(matmul)
    return prediction

# Our model
model = NeuralNetwork()

# Constructing the loss function and the optimization algorithm
criterion = torch.nn.BCELoss(reduction='mean')
optimizer = torch.optim.SGD(model.parameters(), lr=1)

# Step 3: the training process
for epoch in range(10000):

    prediction = model(x)
    loss = criterion(prediction, y)

    if epoch % 1000 == 0 or epoch == 10000 - 1:
        print("epoch ", epoch, ",", "loss: ", loss.item())

    # Backpropagation process
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

and this is it what it gives as an output:

epoch  0 , loss:  0.6983293294906616
epoch  1000 , loss:  0.015215665102005005
epoch  2000 , loss:  0.0048239342868328094
epoch  3000 , loss:  0.00280318153090775
epoch  4000 , loss:  0.001963752554729581
epoch  5000 , loss:  0.0015071843517944217
epoch  6000 , loss:  0.0012211233843117952
epoch  7000 , loss:  0.0010254186345264316
epoch  8000 , loss:  0.000883264874573797
epoch  9000 , loss:  0.0007753585232421756
epoch  9999 , loss:  0.0006908221403136849

As for testing:

# Testing our model
model.eval()

x_test = Variable(torch.Tensor([[1, 1, 0], [0, 0, 1], [0, 1, 1]]))
y_test = Variable(torch.Tensor([[0], [0], [1]]))
y_pred = model(x_test)
print(model(x_test))

with the output:

tensor([[0.0026],
        [0.0011],
        [0.9991]], grad_fn=<SigmoidBackward>)

Output:
To add accuracy you only need one line, namely:

print("Accuracy: ", ((prediction > 0.5) == y).float().mean().item())

When you use sigmoid anything greater than 0.5 is considered positive and anything below negative. (prediction > 0.5) creates a tensor of bool type and you check which of those are equal to y. float() is needed as you cannot calculate mean of bool tensors. item() isn't needed but returns python value from single valued tensor and IMO looks cleaner this way.

You can do the same for test, hence it would be:

model.eval()

x_test = Variable(torch.Tensor([[1, 1, 0], [0, 0, 1], [0, 1, 1]]))
y_test = Variable(torch.Tensor([[0], [0], [1]]))
with torch.no_grad():
    y_pred = model(x_test)
    print("Accuracy: ", ((y_pred > 0.5) == y_test).float().mean().item())

Please notice torch.no_grad(). This context manager disables autograph when you are within it's scope. As you are just passing inputs through your neural network and not training it using gradients there is no need for autograph to be a part of the equation.

Working with logits

It's usually a good habit not to use final activation in your neural networks (unless you really need it). Hence your forward would look like this:

def forward(self, x):
    matmul = self.linear_ij(x)
    activation = torch.sigmoid(matmul)
    # Notice no sigmoid
    return self.linear_jk(activation)

This outputs logits (let's say unnormalized probability ranging from [-inf, inf]) indicating how confident your neural network is it's positive (+inf) or negative class.

You have to change your loss function accordingly, e.g. torch.nn.BCEWithLogitsLoss (mean is deafult reduction, no need to make it explicit here):

criterion = torch.nn.BCEWithLogitsLoss()

Finally, accuracy changes slightly as well. Now anything greater than 0 is considered positive, hence you would do this:

print("Accuracy: ", ((prediction > 0) == y).float().mean().item())

If you need probability you can use torch.sigmoid on the output still, but you might not even need it (as it seems in this case).

EDIT

You should also specify your data as torch.Tensor, torch.Variable is deprecated (tensors already have requires_grad=True), e.g.:

x = torch.Tensor(
    [[0, 0, 1], [0, 1, 1], [1, 0, 1], [0, 1, 0], [1, 0, 0], [1, 1, 1], [0, 0, 0]]
)
y = torch.Tensor([[0], [1], [1], [1], [1], [0], [0]])

EDIT2:

It should be placed below (or above) your loss printing, e.g.:

for epoch in range(10000):

    prediction = model(x)
    loss = criterion(prediction, y)

    if epoch % 1000 == 0 or epoch == 10000 - 1:
        # Here is fine
        print("Accuracy: ", ((prediction > 0.5) == y).float().mean().item())
        print("epoch ", epoch, ",", "loss: ", loss.item())

    # Backpropagation process
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


Instruction: How do I know whether an instance is stored on GPU with PyTorch?

Input:
I'm learning PyTorch recently, and this question comes up.
For example, if I have a net inheriting the "torch.nn.Module".
class Net(torch.nn.Module):
    def __init__(self, something):
        super(net, self).__init__()
        self.p1=something

    def forward():
        pass

net1=Net(123)
net1.cuda()  ##Here I can't see what is changed.

Then how can I know whether net1 (and that something) is stored on GPU.
I've read how the *.cuda() works, seems like let all the "children" run the *.cuda(). I tried to see what the "children" are. It seems the net1 above has no children.

Output:
To check a simple tensor, you can check the is_cuda attribute. For example:
x = torch.zeros(100).cuda()
y = torch.zeros(100)

print(x.is_cuda) # True
print(y.is_cuda) # False

To check a model, a think the easiest way is using the parameters() method, which returns all trainable parameters of your model.
next(model.parameters()).is_cuda


Instruction: How can I convert this keras cnn model to pytorch version

Input:
This is the example keras code that I want to convert to pytorch. My input dataset is 10000*1*102 (two dimensions for labels). The dataset includes 10000 samples. Each sample contains one row with 102 features. I am thinking to use 1dcnn for regression.

PS: hyper-parameter (e.g. filters, kernel_size, stride, padding) could be adjusted based on my 10000*1*102 dataset.

model = Sequential()
model.add(Conv1D(filters=64, kernel_size=3, activation='relu', input_shape=(n_timesteps,n_features)))
model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
model.add(Dropout(0.5))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(100, activation='relu'))
model.add(Dense(n_outputs, activation='softmax'))

Output:
Welcome to pytorch. :)
I am really glad you decide to switch from Keras to PyTorch. It was an important step for me to understand how NNs work in more detail. If you have any specific questions about code or if it isn't working please let me know.

import torch.nn as nn
a0 = nn.Conv1D(n_timesteps, 64, 3)
a1 = nn.Relu()
b0 = nn.Conv1D(64, 64, 3)
b1 = nn.Relu()
c0 = torch.nn.Dropout(p=0.5)
d0 = nn.MaxPool1d(2)
e0 = nn.Flatten()
e1 = nn.Linear(32*n_timesteps,100)
e2 = nn.Relu()
e3 = nn.Linear(n_outputs)
f0 = nn.Softmax(dim=1)

model = nn.Sequential(a0,a1,b0,b1,c0,d0,e0,e1,e2,e3,f0)


Instruction: Pytorch default dataloader gets stuck for large image classification training set

Input:
I am training image classification models in Pytorch and using their default data loader to load my training data. I have a very large training dataset, so usually a couple thousand sample images per class. I've trained models with about 200k images total without issues in the past. However I've found that when have over a million images in total, the Pytorch data loader get stuck.

I believe the code is hanging when I call datasets.ImageFolder(...). When I Ctrl-C, this is consistently the output:

Traceback (most recent call last):                                                                                │
  File "main.py", line 412, in <module>                                                                                │
    main()                                                                                │
  File "main.py", line 122, in main                                                                                │
    run_training(args.group, args.num_classes)                                                                                │
  File "main.py", line 203, in run_training                                                                                │
    train_loader = create_dataloader(traindir, tfm.train_trans, shuffle=True)                                                      │
  File "main.py", line 236, in create_dataloader                                                                                │
    dataset = datasets.ImageFolder(directory, trans)                                                                               │
  File "/home/username/.local/lib/python3.5/site-packages/torchvision/datasets/folder.py", line 209, in __init__     │
    is_valid_file=is_valid_file)                                                                                │
  File "/home/username/.local/lib/python3.5/site-packages/torchvision/datasets/folder.py", line 94, in __init__      │
    samples = make_dataset(self.root, class_to_idx, extensions, is_valid_file)                                                     │
  File "/home/username/.local/lib/python3.5/site-packages/torchvision/datasets/folder.py", line 47, in make_dataset  │
    for root, _, fnames in sorted(os.walk(d)):                                                                                │
  File "/usr/lib/python3.5/os.py", line 380, in walk                                                                               │
    is_dir = entry.is_dir()                                                                                │
Keyboard Interrupt

I thought there might be a deadlock somewhere, however based off the stack output from Ctrl-C it doesn't look like its waiting on a lock. So then I thought that the dataloader was just slow because I was trying to load a lot more data. I let it run for about 2 days and it didn't make any progress, and in the last 2 hours of loading I checked the amount of RAM usage stayed the same.  I also have been able to load training datasets with over 200k images in less than a couple hours in the past. I also tried upgrading my GCP machine to have 32 cores, 4 GPUs, and over 100GB in RAM, however it seems to be that after a certain amount of memory is loaded the data loader just gets stuck.

I'm confused how the data loader could be getting stuck while looping through the directory, and I'm still unsure if its stuck or just extremely slow. Is there some way I can change the Pytortch dataloader to be able to handle 1million+ images for training? Any debugging suggestions are also appreciated!

Thank you!

Output:
It's not a problem with DataLoader, it's a problem with torchvision.datasets.ImageFolder and how it works (and why it works much much worse the more data you have).

It hangs on this line, as indicated by your error:

for root, _, fnames in sorted(os.walk(d)):

Source can be found here.

Underlying problem is it keeps each path and corresponding label in giant list, see the code below (a few things removed for brevity):

def make_dataset(dir, class_to_idx, extensions=None, is_valid_file=None):
    images = []
    dir = os.path.expanduser(dir)
    # Iterate over all subfolders which were found previously
    for target in sorted(class_to_idx.keys()):
        d = os.path.join(dir, target) # Create path to this subfolder
        # Assuming it is directory (which usually is the case)
        for root, _, fnames in sorted(os.walk(d, followlinks=True)):
            # Iterate over ALL files in this subdirectory
            for fname in sorted(fnames):
                path = os.path.join(root, fname)
                # Assuming it is correctly recognized as image file
                item = (path, class_to_idx[target])
                # Add to path with all images
                images.append(item)

    return images

Obviously images will contain 1 million strings (quite lengthy as well) and corresponding int for the classes which definitely is a lot and depends on RAM and CPU.

You can create your own datasets though (provided you change names of your images beforehand) so no memory will be occupied by the dataset.

Setup data structure

Your folder structure should look like this:

root
    class1
    class2
    class3
    ...

Use how many classes you have/need.

Now each class should have the following data:

class1
    0.png
    1.png
    2.png
    ...

Given that you can move on to creating datasets.

Create Datasets

Below torch.utils.data.Dataset uses PIL to open images, you could do it in another way though:

import os
import pathlib

import torch
from PIL import Image

class ImageDataset(torch.utils.data.Dataset):
    def __init__(self, root: str, folder: str, klass: int, extension: str = "png"):
        self._data = pathlib.Path(root) / folder
        self.klass = klass
        self.extension = extension
        # Only calculate once how many files are in this folder
        # Could be passed as argument if you precalculate it somehow
        # e.g. ls | wc -l on Linux
        self._length = sum(1 for entry in os.listdir(self._data))

    def __len__(self):
        # No need to recalculate this value every time
        return self._length

    def __getitem__(self, index):
        # images always follow [0, n-1], so you access them directly
        return Image.open(self._data / "{}.{}".format(str(index), self.extension))

Now you can create your datasets easily (folder structure assumed like the one above:

root = "/path/to/root/with/images"
dataset = (
    ImageDataset(root, "class0", 0)
    + ImageDataset(root, "class1", 1)
    + ImageDataset(root, "class2", 2)
)

You could add as many datasets with specified classes as you wish, do it in loop or whatever.

Finally, use torch.utils.data.DataLoader as per usual, e.g.:

dataloader = torch.utils.data.DataLoader(dataset, batch_size=64, shuffle=True)


Instruction: Size of the training data of GPT2-XL pre-trained model

Input: In huggingface transformer, it is possible to use the pre-trained GPT2-XL language model. But I don't find, on which dataset it is trained? Is it the same trained model which OpenAI used for their paper (trained on 40GB dataset called webtext) ?

Output: The GPT2-XL model is the biggest of the four architectures detailed in the paper you linked (1542M parameters). It is trained on the same data as the other three, which is the WebText you're mentioning.


Instruction: How do I use Pytorch's View function correctly?

Input:
I have a tensor of size x={4,2,C,H,W}. I need it reshape to y={8,C,H,W}, but I want to make sure the images are stored in the right order so say an image at x[1,0,:,:,:] has to be equal to y[2,C,H,W]. I know I can use the view function for this but I am not sure how to use it correctly.

Currently I am doing it as such

feat_imgs_all = feat_imgs_all.view(
    rgb.shape[0], rgb.shape[1], feat_imgs_all.shape[1],
    feat_imgs_all.shape[2], feat_imgs_all.shape[3])

This seems really hacky, is there a way I can just feed the first two shapes, and pytorch figures out the rest?

Output:
You can do it easily using flatten and end_dim argument, see documentation:

import torch

a = torch.randn(4, 2, 32, 64, 64)
flattened = a.flatten(end_dim=1)

torch.all(flattened[2, ...] == a[1, 0, ...]) # True

view could be used as well, like below, though it's not too readable nor too pleasant:

import torch

a = torch.randn(4, 2, 32, 64, 64)
flattened = a.view(-1, *a.shape[2:])

torch.all(flattened[2, ...] == a[1, 0, ...]) # True as well


Instruction: Pytorch Tensorboard SummaryWriter.add_video() Produces Bad Videos

Input:
I’m trying to create a video by generating a sequence of 500 matplotlib plots, converting each to a numpy array, stacking them and then passing them to a SummaryWriter()'s add_video(). When I do this, the colorbar is converted from colored to black & white, and only a small number (~3-4) of the matplotlib plots are repeated. I confirmed that my numpy arrays are correct by using them to recreate a matplotlib figure.

My input tensor has shape (B,C,T,H,W), dtype np.uint8, and values between [0, 255].

Minimal working example below. To be clear, the code runs without any errors. My problem is that the resulting video is wrong.

import matplotlib.pyplot as plt
import numpy as np
import torch
from torch.utils.tensorboard import SummaryWriter

tensorboard_writer = SummaryWriter()
print(tensorboard_writer.get_logdir())

def fig2data(fig):

    # draw the renderer
    fig.canvas.draw()

    # Get the RGB buffer from the figure
    data = np.frombuffer(fig.canvas.tostring_rgb(), dtype=np.uint8)
    data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,))
    return data

size = 500
x = np.random.uniform(0, 2., size=500)
y = np.random.uniform(0, 2., size=500)
trajectory_len = len(x)
trajectory_indices = np.arange(trajectory_len)
width, height = 3, 2

# tensorboard takes video of shape (B,C,T,H,W)
video_array = np.zeros(
    shape=(1, 3, trajectory_len, height*100, width*100),
    dtype=np.uint8)

for trajectory_idx in trajectory_indices:

    fig, axes = plt.subplots(
        1,
        2,
        figsize=(width, height),
        gridspec_kw={'width_ratios': [1, 0.05]})
    fig.suptitle('Example Trajectory')
    # plot the first trajectory
    sc = axes[0].scatter(
        x=[x[trajectory_idx]],
        y=[y[trajectory_idx]],
        c=[trajectory_indices[trajectory_idx]],
        s=4,
        vmin=0,
        vmax=trajectory_len,
        cmap=plt.cm.jet)

    axes[0].set_xlim(-0.25, 2.25)
    axes[0].set_ylim(-0.25, 2.25)

    colorbar = fig.colorbar(sc, cax=axes[1])
    colorbar.set_label('Trajectory Index Number')

    # extract numpy array of figure
    data = fig2data(fig)

    # UNCOMMENT IF YOU WANT TO VERIFY THAT THE NUMPY ARRAY WAS CORRECTLY EXTRACTED
    # plt.show()
    # fig2 = plt.figure()
    # ax2 = fig2.add_subplot(111, frameon=False)
    # ax2.imshow(data)
    # plt.show()

    # close figure to save memory
    plt.close(fig=fig)

    video_array[0, :, trajectory_idx, :, :] = np.transpose(data, (2, 0, 1))

# tensorboard takes video_array of shape (B,C,T,H,W)
tensorboard_writer.add_video(
    tag='sampled_trajectory',
    vid_tensor=torch.from_numpy(video_array),
    global_step=0,
    fps=4)

print('Added video')

tensorboard_writer.close()

Output: According to pytorch docs, the video tensor should have shape (N,T,C,H,W), which I think it means: batch, time, channels, height and width. You said your tensor has shape (B,C,T,H,W). So it seems as your channels and time axis are swapped.


Instruction: How do I serialize an NLP classification PyTorch model

Input:
I am attempting to use a new NLP model within the PyTorch android demo app Demo App Git however I am struggling to serialize the model so that it works with Android.

The demonstration given by PyTorch is as follows for a Resnet model:

model = torchvision.models.resnet18(pretrained=True)
model.eval()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("app/src/main/assets/model.pt")

However I am not sure what to use for the 'example' input with my NLP model.

The model that I am using from a fastai tutorial and the python is linked here: model

Here is the Python used to create my model (using the Fastai library). It is the same as in the model link above, but in a simplified form.

from fastai.text import *
path = untar_data('http://files.fast.ai/data/examples/imdb_sample')
path.ls()
#: [PosixPath('/storage/imdb_sample/texts.csv')]
data_lm = TextDataBunch.from_csv(path, 'texts.csv')
data = (TextList.from_csv(path, 'texts.csv', cols='text')
                .split_from_df(col=2)
                .label_from_df(cols=0)
                .databunch())
bs=48
path = untar_data('https://s3.amazonaws.com/fast-ai-nlp/imdb')
data_lm = (TextList.from_folder(path)
            .filter_by_folder(include=['train', 'test', 'unsup'])
            .split_by_rand_pct(0.1)
            .label_for_lm()
            .databunch(bs=bs))
learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)
learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))
learn.unfreeze()
learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7))
learn.save_encoder('fine_tuned_enc')
path = untar_data('https://s3.amazonaws.com/fast-ai-nlp/imdb')
data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)
             .split_by_folder(valid='test')
             .label_from_folder(classes=['neg', 'pos'])
             .databunch(bs=bs))
learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)
learn.load_encoder('fine_tuned_enc')
learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))
learn.freeze_to(-2)
learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))
learn.freeze_to(-3)
learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))
learn.unfreeze()
learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))

Output:
I worked out how to do this after a while. The issue was that the Fastai model wasn't tracing correctly no matter what shape of input I was using.

In the end, I used another text classification model and got it to work. I wrote a tutorial about how I did it, in case it can help anyone else.

NLP PyTorch Tracing Tutorial

Begin by opening a new Jupyter Python Notebook using your preferred cloud machine provider (I use Paperspace).

Next, copy and run the code in the PyTorch Text Classification tutorial. But replace the line…

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

With…

device = torch.device("cpu")

NOTE: It caused issues tracing when the device was set to CUDA so I forced it on to the CPU. (this will slow training, but inference on the mobile will run at the same speed as it is cpu anyway)

Lastly, run the code below to correctly trace the model to allow it to be run on Android:

data = DataLoader(test_dataset, batch_size=1, collate_fn=generate_batch)
for text, offsets, cls in data:
        text, offsets, cls = text.to(device), offsets.to(device), cls.to(device)
example = text, offsets
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt")

In addition, if you would like a CSV copy of the vocab list for use on Android when you are making predictions, run the following code afterwards:

import pandas as pd
vocab = train_dataset.get_vocab()
df = pd.DataFrame.from_dict(vocab.stoi, orient='index', columns=['token'])
df[:30]
df.to_csv('out.csv')

This model should work fine on Android using the PyTorch API.


Instruction: How to get the predict probability?

Input:
This code get the 1 or 0 value from model.
If I want to get the probability of the prediction
Which line should I change?

from torch.autograd import Variable
results = []
#names = []
with torch.no_grad():
    model.eval()
    print('===============================================start')
    for num, data in enumerate(test_loader):
        #print(num)
        print("=====================================================")

        imgs, label = data
        imgs,labels = imgs.to(device), label.to(device)
        test = Variable(imgs)
        output = model(test)
        #print(output)
        ps = torch.exp(output)
        print(ps)
        top_p, top_class = ps.topk(1, dim = 1)
        results += top_class.cpu().numpy().tolist()

model = models.resnet50(pretrained=True)
model.fc = nn.Linear(2048, num_classes)
model.cuda()

Output:
Models usually outputs raw prediction logits. To convert them to probability you should use softmax function
import torch.nn.functional as nnf

# ...
prob = nnf.softmax(output, dim=1)

top_p, top_class = prob.topk(1, dim = 1)

new variable top_p should give you the probability of the top k classes.


Instruction: How to check the root cause of CUDA out of memory issue in the middle of training?

Input: I'm running roberta on huggingface language_modeling.py. After doing 400 steps I suddenly get a CUDA out of memory issue. Don't know how to deal with it. Can you please help? Thanks

Output:
My problem was that I didn't check the size of my GPU memory with comparison to the sizes of samples. I had a lot of pretty small samples and after many iterations a large one. My bad.
Thank you and remember to check these things if it happens to you to.


Instruction: How to embed Sequence of Sentences in RNN?

Input:
I am trying to make a RNN model (in Pytorch), that takes couple of sentences and then classifies it to be either Class 0 or Class 1.

For the sake of this question let's assume that the max_len of the sentence is 4 and max_amount of time steps is 5. Thus, each datapoint is on the form (0  is a value that used for padding padded value):

    x[1] = [
    # Input features at timestep 1
    [1, 48, 91, 0],
    # Input features at timestep 2
    [20, 5, 17, 32],
    # Input features at timestep 3
    [12, 18, 0, 0],
    # Input features at timestep 4
    [0, 0, 0, 0],
    # Input features at timestep 5
    [0, 0, 0, 0]
    ]
    y[1] = [1]

When I have just one sentence per target: I simply pass each word to the embedding layer and then to the LSTM or GRU, but I am a bit stuck on what to do when I have a sequence of sentences per target?

How do I build an embedding that can handle sentences?

Output:
The simplest way is to use 2 kinds of LSTM.

Prepare the toy dataset

xi = [
# Input features at timestep 1
[1, 48, 91, 0],
# Input features at timestep 2
[20, 5, 17, 32],
# Input features at timestep 3
[12, 18, 0, 0],
# Input features at timestep 4
[0, 0, 0, 0],
# Input features at timestep 5
[0, 0, 0, 0]
]
yi = 1

x = torch.tensor([xi, xi])
y = torch.tensor([yi, yi])

print(x.shape)
# torch.Size([2, 5, 4])

print(y.shape)
# torch.Size([2])

Then, x is the batch of inputs. Here batch_size = 2.

Embed the input

vocab_size = 1000
embed_size = 100
hidden_size = 200
embed = nn.Embedding(vocab_size, embed_size)

# shape [2, 5, 4, 100]
x = embed(x)

The first word-LSTM is to encode each sequence into a vector

# convert x into a batch of sequences
# Reshape into [2, 20, 100]
x = x.view(bs * 5, 4, 100)

wlstm = nn.LSTM(embed_size, hidden_size, batch_first=True)
# get the only final hidden state of each sequence

_, (hn, _) = wlstm(x)

# hn shape [1, 10, 200]

# get the output of final layer
hn = hn[0] # [10, 200]

The second seq-LSTM is to encode sequences into a single vector

# Reshape hn into [bs, num_seq, hidden_size]
hn = hn.view(2, 5, 200)

# Pass to another LSTM and get the final state hn
slstm = nn.LSTM(hidden_size, hidden_size, batch_first=True)
_, (hn, _) = slstm(hn) # [1, 2, 200]

# Similarly, get the hidden state of the last layer
hn = hn[0] # [2, 200]

Add some classification layers

pred_linear = nn.Linear(hidden_size, 1)

# [2, 1]
output = torch.sigmoid(pred_linear(hn))


Instruction: TypeError: 'tuple' object cannot be interpreted as an integer while creating data generators PyTorch

Input:
I am new to PyTorch and I am learning to create batches of data for segmentation. The code is shown below:

class NumbersDataset(Dataset):
    def __init__(self):
        self.X = list(df['input_img'])
        self.y = list(df['mask_img'])

    def __len__(self):
        return len(self.X), len(self.y)

    def __getitem__(self, idx):
        return self.X[idx], self.y[idx]

if __name__ == '__main__':
    dataset = NumbersDataset()
    dataloader = DataLoader(dataset, batch_size=50, shuffle=True, num_workers=2)
    # print(len(dataset))
    # plt.imshow(dataset[100])
    # plt.show()
    print(next(iter(dataloader)))

where df['input_img'] column contains the location of the image ('/path/to/pic/480p/boxing-fisheye/00010.jpg') and df['mask_img'] contains the location of all the mask images. I am trying to load the images but I get the error:

  TypeError: 'tuple' object cannot be interpreted as an integer

However, if I don't use DataLoader and just do the following:

     dataset = NumbersDataset()
     print(len(dataset))
     print(dataset[10:20])

then I get what I expect. Can someone tell me what am I doing wrong?

Output:
You cannot return a tuple for the __len__ method.  The expected type is int

# perhaps you can add the list length's for the total length
# but no matter how you choose to implement the method you can
# only return on value of type integer `int`
def __len__(self):
    return len(self.X) + len(self.y)


Instruction: Why is very simple PyTorch LSTM model not learning?

Input:
I am trying to do very simple learning so that I can better understand how PyTorch and LSTMs work. To that end, I am trying to learn a mapping from an input tensor to an output tensor (same shape) that is twice the value. So [1 2 3] as input should learn [2 4 6] as an output. To that end, I have a dataloader:

class AudioDataset(Dataset):
    def __init__(self, corrupted_path, train_set=False, test_set=False):
        torch.manual_seed(0)
        numpy.random.seed(0)

    def __len__(self):
        return len(self.file_paths)

    def __getitem__(self, index):
        random_tensor = torch.rand(1, 5) * 2
        random_tensor = random_tensor - 1

        return random_tensor, random_tensor * 2

My LSTM itself is pretty simple:

class MyLSTM(nn.Module):
    def __init__(self, input_size=4000):
        super(MyLSTM, self).__init__()

        self.lstm = nn.LSTM(input_size=input_size, hidden_size=input_size,
                            num_layers=2)

    def forward(self, x):
        y = self.lstm(x)
        return y

My training looks like:

    train_loader = torch.utils.data.DataLoader(
        train_set, batch_size=1, shuffle=True, **kwargs)

    model = MyLSTM(input_size=5)
    optimizer = optim.Adam(model.parameters(), lr=0.01, weight_decay=0.0001)
    loss_fn = torch.nn.MSELoss(reduction='sum')

    for epoch in range(300):
        for i, data in enumerate(train_loader):
            inputs = data[0]
            outputs = data[1]

            print('inputs', inputs, inputs.size())
            print('outputs', outputs, outputs.size())
            optimizer.zero_grad()

            pred = model(inputs)
            print('pred', pred[0], pred[0].size())

            loss = loss_fn(pred[0], outputs)

            model.zero_grad()

            loss.backward()
            optimizer.step()

After 300 epochs, my loss looks like tensor(1.4892, grad_fn=<MseLossBackward>). Which doesn't appear to be very good.

Randomly looking at some of the inputs / outputs and predictions:

inputs tensor([[[0.5050, 0.4669, 0.8310,  ..., 0.0659, 0.5043, 0.8885]]]) torch.Size([1, 1, 4000])
outputs tensor([[[1.0100, 0.9338, 1.6620,  ..., 0.1319, 1.0085, 1.7770]]]) torch.Size([1, 1, 4000])
pred tensor([[[ 0.6930,  0.0231, -0.6874,  ..., -0.5225,  0.1096,  0.5796]]],
       grad_fn=<StackBackward>) torch.Size([1, 1, 4000])

We see that it hasn't learned very much at all. I can't understand what I'm doing wrong; if someone can guide me, that would be greatly appreciated.

Output:
LSTMs are made of neurons that generate an internal state based upon a feedback loop from previous training data. Each neuron has four internal gates that take multiple inputs and generate multiple outputs. It's one of the more complex neurons to work with and understand, and I'm not really skilled enough to give an in-depth answer.

What I see in your example code is a lack of understanding of how they work, and it seems like you're assuming they work like a linear layer. I say that, because your forward method doesn't handle the internal state and you're not reshaping the outputs.

You define the LSTM like this:

     self.lstm = nn.LSTM(input_size=input_size, hidden_size=input_size, num_layers=2)

The hidden_size relates to how memory and features work with the gates.

PyTorch documentation says the following:

  hidden_size – The number of features in the hidden state h

It is referring to the size of the hidden state used to train the internal gates for long and short term memory. The gates are a function across the hidden features that store previous gate outputs. Each time a neuron is trained the hidden state is updated and is used again for the next training data.

So why is this so important?

You are throwing away the hidden state data during training, and I don't know what happens if you don't define the hidden state. I assume the LSTM works as if there is never any history.

The forward function should look something like this:

    def forward(self, x, hidden):
        lstm_output, hidden = self.lstm(x, hidden)
        return lstm_output, hidden

During training you have to keep track of the hidden state yourself.

for i in range(epochs):
   hidden = (torch.zeros(num_layers, batch_size, num_hidden),
             torch.zeros(num_layers, batch_size, num_hidden))

   for x, y in generate_batches(...):
        # missing code....
        lstm_output, hidden = model.forward(x, hidden)

Take note of the shape for the hidden state. It's different from what you usually do with linear layers.

There are some steps above missing that relate to resetting the hidden state, but I can't remember how that part works.

LSTMs on their own only describe features much like convolution layers. It's unlikely that the outputs from a LSTMs is what you're interested in using.

Most models that use LSTMs or convolutions will have a bottom section of fully connected layers (for example: nn.Linear()). These layers will train on the features to predict the outputs you're interested in.

The problem here is that the outputs from LSTMs are in the wrong shape, and you have to reshape the tensors so that a linear layer can use them.

Here is an example LSTM forward function that I have used:

    def forward(self, x, hidden):
        lstm_output, hidden = self.lstm(x, hidden)
        drop_output = self.dropout(lstm_output)
        drop_output = drop_output.contiguous().view(-1, self.num_hidden)
        final_out = self.fc_linear(drop_output)
        return final_out, hidden

LSTMs are definitely an advanced topic in machine learning, and PyTorch isn't an easy library to learn to begin with. I would recommend reading up on LSTMs using the TensorFlow documentation and online blogs to get a better grasp of how they work.


Instruction: Advance indexing in Pytorch to get rid of nested for-loops

Input:
I have a situation for which I am using nested for-loops, but I want to know if there's a faster way of doing this using some advanced indexing in Pytorch.

I have a tensor named t:

t = torch.randn(3,8)
print(t)
tensor([[-1.1258, -1.1524, -0.2506, -0.4339,  0.8487,  0.6920, -0.3160, -2.1152],
        [ 0.4681, -0.1577,  1.4437,  0.2660,  0.1665,  0.8744, -0.1435, -0.1116],
        [ 0.9318,  1.2590,  2.0050,  0.0537,  0.6181, -0.4128, -0.8411, -2.3160]])

I want to create a new tensor which indexes values from t.
Let's say these indexes are stored in variable indexes

indexes = [[(0, 1, 4, 5), (0, 1, 6, 7), (4, 5, 6, 7)],
           [(2, 3, 4, 5)],
           [(4, 5, 6, 7), (2, 3, 6, 7)]]

Each inner tuple in indexes represents four indexes that are to be taken from a row.

As an example, based on these indexes my output would be a 6x4 dimension tensor (6 is the total number of tuples in indexes, and 4 corresponds to one value in a tuple)

For instance, this is what I want to do:

#counting the number of tuples in indexes
count_instances = sum([1 for lst in indexes for tupl in lst])

#creating a zero output matrix
final_tensor = torch.zeros(count_instances,4)

final_tensor[0] = t[0,indexes[0][0]]
final_tensor[1] = t[0,indexes[0][1]]
final_tensor[2] = t[0,indexes[0][2]]
final_tensor[3] = t[1,indexes[1][0]]
final_tensor[4] = t[2,indexes[2][0]]
final_tensor[5] = t[2,indexes[2][1]]

The final output looks like this:
print(final_tensor)

tensor([[-1.1258, -1.1524,  0.8487,  0.6920],
        [-1.1258, -1.1524, -0.3160, -2.1152],
        [ 0.8487,  0.6920, -0.3160, -2.1152],
        [ 1.4437,  0.2660,  0.1665,  0.8744],
        [ 0.6181, -0.4128, -0.8411, -2.3160],
        [ 2.0050,  0.0537, -0.8411, -2.3160]])

I created a function build_tensor (shown below) to achieve this with nested for-loops, but I want to know if there's a faster way of doing it with simple indexing in Pytorch. I want a faster way of doing it because I'm doing this operation hundreds of times with bigger index and t sizes.

Any help?

def build_tensor(indexes, t):
    #count tuples
    count_instances = sum([1 for lst in indexes for tupl in lst])
    #create a zero tensor
    final_tensor = torch.zeros(count_instances,4)
    final_tensor_idx = 0

    for curr_idx, lst in enumerate(indexes):
        for tupl in lst:
            final_tensor[final_tensor_idx] = t[curr_idx,tupl]
            final_tensor_idx+=1
    return final_tensor

Output:
You can arrange the indices into 2D arrays then do the indexing in one shot like this:

rows = [(row,)*len(index_tuple) for row, row_indices in enumerate(indexes) for index_tuple in row_indices]
columns = [index_tuple for row_indices in indexes for index_tuple in row_indices]
final_tensor = t[rows, columns]


Instruction: Pytorch summary only works for one specific input size for U-Net

Input:
I am trying to implement the UNet architecture in Pytorch. When I print the model using print(model) I get the correct architecture:

but when I try to print the summary using (or any other input size for that matter):

from torchsummary import summary
summary(model, input_size=(13, 572, 572))

I get an error:

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 70 and 71 in dimension 2 at /Users/distiller/project/conda/conda-bld/pytorch_1579022061893/work/aten/src/TH/generic/THTensor.cpp:612

However, it works perfectly if I give the input_size as input_size=(3, 224, 224))( like it worked for this person here). I am so baffled.

Can someone help me what's wrong?

Edit: I have used the model architecture from here.

Output:
This UNet architecture you provided doesn't support that shape (unless the depth parameter is <= 3). Ultimately the reason for this is that the size of a downsampling operation isn't invertible since multiple input shapes map to the same output shape. For example consider

>> torch.nn.functional.max_pool2d(torch.zeros(1, 1, 10, 10), 2).shape
torch.Size([1, 1, 5, 5])
>> torch.nn.functional.max_pool2d(torch.zeros(1, 1, 11, 11), 2).shape
torch.Size([1, 1, 5, 5])

So the question is, given only the output shape is 5x5, what was the shape of the input? Was it 10x10 or 11x11? This same phenomenon applies to downsampling via strided convolutions.

The problem is that the UNet class tries to combine features from the downsampling half to the network to the features in the upsampling half. If it "guesses wrong" about the original shape during upsampling then you will receive a dimension mismatch error.

To avoid this issue you'll need to ensure that the height and width of your input data are multiples of 2**(depth-1). So, for the default depth=5 you need the input image height and width to be a multiple of 16 (e.g. 560 or 576). Alternatively, since 572 is divisible by 4 then you could also set depth=3 to make it work.


Instruction: Pysyft Federated learning, Error with Websockets

Input:
I am trying to run a federated learning from pysyft (https://github.com/OpenMined/PySyft/blob/dev/examples/tutorials/advanced/websockets-example-MNIST-parallel/Asynchronous-federated-learning-on-MNIST.ipynb) that creates remote workers and connect to them via websockets. however I am getting an error in folllowing evaluation step.

future: <Task finished coro=<WebsocketServerWorker._producer_handler() done, defined at C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py:95> exception=AttributeError("'dict' object has no attribute 'owner'")>
Traceback (most recent call last):
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 663, in register_response
    register_response_function = register_response_functions[attr_id]
KeyError: 'evaluate'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py", line 113, in _producer_handler
    response = self._recv_msg(message)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py", line 124, in _recv_msg
    return self.recv_msg(message)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\base.py", line 310, in recv_msg
    response = self._message_router[type(msg)](msg.contents)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\base.py", line 457, in execute_command
    command_name, response, list(return_ids), self
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 672, in register_response
    new_response = register_response_function(response, response_ids=response_ids, owner=owner)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 766, in <lambda>
    return lambda x, **kwargs: f(lambdas, x, **kwargs)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 522, in two_fold
    return lambdas[0](args[0], **kwargs), lambdas[1](args[1], **kwargs)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 744, in <lambda>
    else lambda i, **kwargs: register_tensor(i, **kwargs)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 712, in register_tensor
    tensor.owner = owner
AttributeError: 'dict' object has no attribute 'owner'

There are no clear answer from their forum. does anyone have any clue as to what the issue is in this script.

My syft version:

syft : 0.2.3a1
syft-proto : 0.1.1a1.post12
torch : 1.4.0

Output: I came across this problem as well and pushed a fix in https://github.com/OpenMined/PySyft/pull/2948


Instruction: Why pytorch model cannot recognize the tensors I defined?

Input:
I'm just learn pytorch recently.
And I try to write a same model like the paper that I have read for practice.
This is the PDF of the paper I refer.
https://dl.acm.org/doi/pdf/10.1145/3178876.3186066?download=true
Here is the code what I wrote.
class Tem(torch.nn.Module):
    def __init__(self, embedding_size, hidden_size):
        super(Tem, self).__init()
        self.embedding_size = embedding_size
        self.hidden_size = hidden_size
        self.leaf_size = 0

        self.xgb_model = None
        self.vec_embedding = None

        self.multi_hot_Q = None
        self.user_embedding = torch.nn.Linear(1, embedding_size)
        self.item_embedding = torch.nn.Linear(1, embedding_size)

    def pretrain(self, ui_attributes, labels):
        print("Start XGBoost Training...")
        self.xgb_model = XGBoost(ui_attributes, labels)
        self.leaf_size = self.xgb_model.leaf_size
        self.vec_embedding = Variable(torch.rand(self.embedding_size, self.leaf_size, requires_grad=True))

        self.h = Variable(torch.rand(self.hidden_size, 1, requires_grad=True))
        self.att_w = Variable(torch.rand(2 * self.embedding_size, self.hidden_size, requires_grad=True))
        self.att_b = Variable(torch.rand(self.leaf_size, self.hidden_size, requires_grad=True))

        self.r_1 = Variable(torch.rand(self.embedding_size, 1, requires_grad=True))
        self.r_2 = Variable(torch.rand(self.embedding_size, 1, requires_grad=True))
        self.bias = Variable(torch.rand(1, 1, requires_grad=True))

    def forward(self, ui_ids, ui_attributes):
        if self.xgb_model == None:
            raise Exception("Please run Tem.pretrain() to pre-train XGBoost model first.")

        n_data = len(ui_ids)

        att_input = torch.FloatTensor(ui_attributes)
        self.multi_hot_Q = torch.FloatTensor(self.xgb_model.multi_hot(att_input)).permute(0,2,1)

        vq = self.vec_embedding * self.multi_hot_Q

        id_input = torch.FloatTensor(ui_ids)

        user_embedded = self.user_embedding(id_input[:,0].reshape(n_data, 1))

        item_embedded = self.item_embedding(id_input[:,1].reshape(n_data, 1))

        ui = (user_embedded * item_embedded).reshape(n_data, self.embedding_size, 1)

        ui_repeat = ui.repeat(1, 1, self.leaf_size)

        cross = torch.cat([ui_repeat, vq], dim=1).permute(0,2,1)

        re_cross = corss.reshape(cross.shape[0] * cross.shape[1], cross.shape[2])

        attention = torch.mm(re_cross, self.att_w)
        attention = F.leaky_relu(attention + self.att_b.repeat(n_data, 1))
        attention = torch.mm(attention, self.h).reshape(n_data, self.leaf_size)
        attention = F.softmax(attention).reshape(n_data, self.leaf_size, 1)
        attention = self.vec_embedding.permute(1,0) * attention.repeat(1,1,20)

        pool = torch.max(attention, 1).values

        y_hat = self.bias.repeat(n_data, 1) + torch.mm(ui.reshape(n_data, self.embedding_size), self.r_1) + torch.mm(pool, self.r_2)
        y_hat = F.softmax(torch.nn.Linear(1, 2)(y_hat))
        return y_hat

My question is...It seems torch didn't know what tensor should be calculate gradient in backward propagation.
print(tem)

Tem(
(user_embedding): Linear(in_features=1, out_features=20, bias=True)
(item_embedding): Linear(in_features=1, out_features=20, bias=True)
)

I googled this problem, someone says those tensors should use torch.autograd.Variable(), but it didn't solve my problem. And someone says autograd directly supports tensors now. torch.autograd.Variable() is not necessary.
loss_func = torch.nn.CrossEntropyLoss()
optimizer = torch.Adagrad(tem.parameters(), lr=0.02)
for t in range(20):
    prediction = tem(ids_train, att_train)

    loss = loss_func(prediction, y_train)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if t % 5 == 0:
        print("loss: ", loss)

loss: tensor(0.8133, grad_fn=<NllLossBackward>)
loss: tensor(0.8133, grad_fn=<NllLossBackward>)
loss: tensor(0.8133, grad_fn=<NllLossBackward>)
loss: tensor(0.8133, grad_fn=<NllLossBackward>)

Output:
Your problem is not related to Variable. As you said, it's not necessary anymore. To compute the gradients of a tensor declared in a model (that extends nn.Module) you need to include them into the model's parameters using the method nn.Parameter(). For example, to include self.h, you can do:
self.h = nn.Parameter(torch.zeros(10,10)

Now, when you call loss.backward() it'll collect the gradient for this variable (of course, loss must be dependent on self.h).


Instruction: Pytorch transforms.RandomRotation() does not work on Google Colab

Input:
Normally i was working on letter&digit recognition on my computer and I wanted to move my project to Colab but unfortunately there was an error (you can see the error below).
after some debugging i found which line is giving me error.

transforms.RandomRotation(degrees=(90, -90))

below i wrote simple abstract code to show this error.This code does not work on colab but it works fine at my own computer environment.Problem might be about the different versions of pytorch library i have version 1.3.1 on my computer and colab uses version 1.4.0.

import torch
import torchvision
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
    transformOpt = transforms.Compose([
            transforms.RandomRotation(degrees=(90, -90)),
            transforms.ToTensor()
        ])

    train_set = datasets.MNIST(
        root='', train=True, transform=transformOpt, download=True)
    test_set = datasets.MNIST(
        root='', train=False, transform=transformOpt, download=True)

    train_loader = torch.utils.data.DataLoader(
        dataset=train_set,
        batch_size=100,
        shuffle=True)
    test_loader = torch.utils.data.DataLoader(
        dataset=test_set,
        batch_size=100,
        shuffle=False)

    images, labels = next(iter(train_loader))
    plt.imshow(images[0].view(28, 28), cmap="gray")
    plt.show()

  The full error I got when I execute this sample code above on Google Colab.

TypeError                                 Traceback (most recent call last)

<ipython-input-1-8409db422154> in <module>()
     24     shuffle=False)
     25
---> 26 images, labels = next(iter(train_loader))
     27 plt.imshow(images[0].view(28, 28), cmap="gray")
     28 plt.show()

10 frames

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.6/dist-packages/torchvision/datasets/mnist.py in __getitem__(self, index)
     95
     96         if self.transform is not None:
---> 97             img = self.transform(img)
     98
     99         if self.target_transform is not None:

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py in __call__(self, img)
     68     def __call__(self, img):
     69         for t in self.transforms:
---> 70             img = t(img)
     71         return img
     72

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py in __call__(self, img)    1001         angle = self.get_params(self.degrees)    1002
-> 1003         return F.rotate(img, angle, self.resample, self.expand, self.center, self.fill)    1004     1005     def
__repr__(self):

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py in rotate(img, angle, resample, expand, center, fill)
    727         fill = tuple([fill] * 3)
    728
--> 729     return img.rotate(angle, resample, expand, center, fillcolor=fill)
    730
    731

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in rotate(self, angle, resample, expand, center, translate, fillcolor)    2003         w, h = nw, nh    2004
-> 2005         return self.transform((w, h), AFFINE, matrix, resample, fillcolor=fillcolor)    2006     2007     def save(self,    fp, format=None, **params):

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in transform(self, size, method, data, resample, fill, fillcolor)    2297             raise ValueError("missing method data")    2298
-> 2299         im = new(self.mode, size, fillcolor)    2300         if method == MESH:    2301             # list of quads

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in new(mode, size, color)    2503         im.palette = ImagePalette.ImagePalette()    2504         color = im.palette.getcolor(color)
-> 2505     return im._new(core.fill(mode, size, color))    2506     2507

TypeError: function takes exactly 1 argument (3 given)

Output:
You're absolutely correct. torchvision 0.5 has a bug in RandomRotation() in the fill argument probably due to incompatible Pillow version. This issue has now been fixed (PR#1760) and will be resolved in the next release.

Temporarily, you add fill=(0,) to RandomRotation transform to fix it.

transforms.RandomRotation(degrees=(90, -90), fill=(0,))


Instruction: FileNotFoundError in ...\.cache\ using SentenceTransformer

Input:
Does anyone have experience working with SentenceTransformer (Bert)?

My Code:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('roberta-large-nli-stsb-mean-tokens')

My Error:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\ga2943/.cache\\torch\\sentence_transformers\\public.ukp.informatik.tu-darmstadt.de_reimers_sentence-transformers_v0.2_roberta-large-nli-stsb-mean-tokens.zip\\modules.json'

Once in a while I get a different error running the same Code with another Bert-model:

PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: 'C:\\Users\\ga2943/.cache\\torch\\sentence_transformers\\public.ukp.informatik.tu-darmstadt.de_reimers_sentence-transformers_v0.2_bert-large-nli-cls-token.zip\\model.zip'

(Translates to: Data is beeing used in a different Process at the time, which cant be the case. I restarted my machine to ensure all programs are closed in the background but no difference)

The first time I ran my code it worked. It downloaded the Sentenceencoder I wanted (different Bert model)  Now that I want to test a different model, it does not download it but gives me the errors.

Funny thing is, on a different Computer I was able to load a different model but it resulted in the same case. Only the first ever downloaded model works on the machine.

Any ideas? Still getting the error, no one knows how to fix it?

Output:
I had to re-install everything over again to solve the issue. I cant put my finger on what caused the error. As I did not delete or update anything myself I am assuming anaconda patched something without letting me know.

The second error still remains and occurs unpredictable from time to time.


Instruction: Use of scheduler with self-ajusting optimizers in PyTorch

Input: In PyTorch, the weight adjustment policy is determined by the optimizer, and the learning rate is adjusted with a scheduler. When the optimizer is SGD, there is only one learning rate and this is straightforward. When using Adagrad, Adam, or any similar optimizer which inherently adjusts the learning rate on a per-parameter basis, is there something in particular to look out for? Can I ignore the scheduler completely since the algorithm adjusts its own learning rates? Should I parameterize it very differently than if I'm using SGD?

Output:
The learning rate you define for optimizers like ADAM are upper bounds. You can see this in the paper in Section 2.1. The stepsize α in the paper is the learning rate.

  The effective magnitude of the steps taken in parameter space at each are approximately bounded by the stepsize setting α

Also this stepsize α is directly used and multiplied with the step size correction, which is learned. So changing the learning rate e.g. reducing it will reduce all individual learning rates and reduces the upper bound. This can be helpful during the "end" of an training, to reduce the overall step sizes, so only smaller steps occur and might help the network to find a minima in the loss function.

I saw learning rate decay in some papers using ADAM and used it myself and it did help. What I found is that you should do it slower than e.g. with SGD. With one model I just multiply it with 0.8 every 10 epochs. So it is a gradual decay which I think works better than more drastic steps since you don't "invalidate" the estimated momentums to much. But this is just my theory.


Instruction: Pytorch torch.cholesky ignoring exception

Input:
For some matrices on my batch I'm having an exception due the matrix being singular.

L = th.cholesky(Xt.bmm(X))

  cholesky_cpu: For batch 51100: U(22,22) is zero, singular U

Since they are few for my use case I would like to ignore the exception and further deal with them. I will set the resulting calculation as nan is it possible somehow?

Actually if I catch the exception and use continue still it doesn’t finish the calculation of the rest of the batch.

The same happens in C++ with Pytorch libtorch.

Output:
It's not possible to catch the exception according to Pytorch Discuss forum.

The solution, unfortunately, was to implement my own  simple batched cholesky (th.cholesky(..., upper=False)) and then deal with Nan values using th.isnan.

import torch as th

# nograd cholesky
def cholesky(A):
    L = th.zeros_like(A)

    for i in range(A.shape[-1]):
        for j in range(i+1):
            s = 0.0
            for k in range(j):
                s = s + L[...,i,k] * L[...,j,k]

            L[...,i,j] = th.sqrt(A[...,i,i] - s) if (i == j) else \
                      (1.0 / L[...,j,j] * (A[...,i,j] - s))
    return L


Instruction: PyTorch tensor indexing or conditional selection?

Input:
    for c in range(self.n_class):
        target[c][label == c] = 1

self.n_class is 32. and target is 32 x 1024 x 2048 tensor.

I know that target[c] select the each one of 1 x 1024 x 2048.
But I don't understand [label == c].

Because by thumb of rule, an integer should go in the square [].

Could someone explain what second square does and how it makes sense?

Output:
PyTorch supports "Advanced Indexing." It implements the ability to accept a tensor argument to the [] operator.

The result of the == operator is a boolean mask. The [] operator is using that mask to select elements. This example below might help clarify:

>>> x=torch.arange(0,10)
>>> x
tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> x < 5
tensor([ True,  True,  True,  True,  True, False, False, False, False, False])
>>> x[x < 5]
tensor([0, 1, 2, 3, 4])
>>> x[x > 5]
tensor([6, 7, 8, 9])
>>>

Some general docs:
https://www.pythonlikeyoumeanit.com/Module3_IntroducingNumpy/BasicIndexing.html

Advanced indexing in numpy:
https://numpy.org/doc/1.18/reference/arrays.indexing.html


Instruction: How can I implement basic question answering with hugging-face?

Input:
I have:

from transformers import XLNetTokenizer, XLNetForQuestionAnswering
import torch

tokenizer =  XLNetTokenizer.from_pretrained('xlnet-base-cased')
model = XLNetForQuestionAnswering.from_pretrained('xlnet-base-cased')

input_ids = torch.tensor(tokenizer.encode("What is my name?", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
start_positions = torch.tensor([1])
end_positions = torch.tensor([3])
outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions)
loss = outputs[0]

print(outputs)
print(loss)

As per the docs. This does something giving:

(tensor(2.3008, grad_fn=<DivBackward0>),)
tensor(2.3008, grad_fn=<DivBackward0>)

However, I want an actual answer if possible?

Output:
Thanks to Joe Davison for providing the answer on Twitter:

from transformers import pipeline

qa = pipeline('question-answering')
response = qa(context='I like to eat apples, but hate bananas.',
              question='What do I like?')

print(response)

gives a response of:

{'score': 0.282511100858045, 'start': 31, 'end': 38, 'answer': 'bananas.'}

Not quite right, but at least the score is low.


Instruction: Too slow first run TorchScript model and its implementation in Flask

Input:
I'm trying to deploy torchscripted model in Python and Flask. As I realized (at least as mentioned here ) that scripted models need to be "warmed up" before using, so first run of such models takes much longer than subsequent ones. My question is: is there any way to load torchscripted models in Flask route and predict without loss of "worm-up" time? Can I store somewhere "warm-uped" model to avoid warming-up in every request?
I wrote simple code that reproduce the "warm-up" pass:

    import torchvision, torch, time
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
    model = torch.jit.script(model)
    model.eval()
    x = [torch.randn((3,224,224))]
    for i in range(3):
      start = time.time()
      model(x)
      print(‘Time elapsed: {}’.format(time.time()-start))

Output:

Time elapsed: 38.29

Time elapsed: 6.65

Time elapsed: 6.65

And Flask code:

    import torch, torchvision, os, time
    from flask import  Flask
    app = Flask(__name__)
    @app.route('/')
    def test_scripted_model(path='/tmp/scripted_model.pth'):
        if os.path.exists(path):
            model = torch.jit.load(path, map_location='cpu')
        else:
            model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
            model = torch.jit.script(model)
            torch.jit.save(model, path)
        model.eval()
        x = [torch.randn((3, 224, 224))]
        out = ''
        for i in range(3):
            start = time.time()
            model(x)
            out += 'Run {} time: {};\t'.format(i+1, round((time.time() - start), 2))
        return out
    if __name__ == '__main__':
        app.run(host='0.0.0.0', port=5000, debug=False)

Output:

Run 1 time: 46.01; Run 2 time: 8.76; Run 3 time: 8.55;

OS: Ubuntu 18.04 & Windows10
Python version: 3.6.9
Flask: 1.1.1
Torch: 1.4.0
Torchvision: 0.5.0

Update:

Solved "warm-up" problem as:

with torch.jit.optimized_execution(False):
    model(x)

Update2:
Solved Flask problem (as mentioned below) with creating global python model object before server starts and warming up it there. Then in each request the model is ready to use.

    model = torch.jit.load(path, map_location='cpu').eval()
    model(x)
    app = Flask(__name__)

and then in @app.route:

@app.route('/')
def test_scripted_model():
    global model
    ...
    ...

Output:
  Can I store somewhere "warm-uped" model to avoid warming-up in every request?

Yes, just instantiate your model outside of the test_scripted_model function and refer to it from within the function.


Instruction: How to make a Truncated normal distribution in pytorch?

Input:
I want to create a Truncated normal distribution(that is Gaussian distribution with a range) in PyTorch.
I want to be able to change the mean, std, and range.
Is there a PyTorch method for that?

Output:
Use torch.nn.init.trunc_normal_.
Description as given Here:

Fills the input Tensor with values drawn from a truncated
normal distribution. The values are effectively drawn from the
normal distribution :math:\mathcal{N}(\text{mean}, \text{std}^2)
with values outside :math:[a, b] redrawn until they are within
the bounds. The method used for generating the random values works
best when :math:a \leq \text{mean} \leq b.


Instruction: PyTorch Geometric CUDA installation issues on Google Colab

Input:
I was working on a PyTorch Geometric project using Google Colab for CUDA support. Since it's library isn't present by default, I run:

!pip install --upgrade torch-scatter
!pip install --upgrade torch-sparse
!pip install --upgrade torch-cluster
!pip install --upgrade torch-spline-conv
!pip install torch-geometric

Recently, while importing torch_geometric, owing to version upgrades, there's a CUDA version mismatch saying:

  RuntimeError: Detected that PyTorch and torch_sparse were compiled with different CUDA versions. PyTorch has CUDA version 10.1 and torch_sparse has CUDA version 10.0. Please reinstall the torch_sparse that matches your PyTorch install.

To solve this, I tried using conda for specific CUDA version as:

!conda install pytorch==1.4.0 cudatoolkit=10.0 -c pytorch

Yet, on running print(torch.version.cuda), I get 10.1 as the output and not 10.0 as I wanted.

This is a recent error since it wasn't throwing up this issue in past week. Any best practice to solve this issue?

Output:
From their website

Try this

!pip install torch-geometric \
  torch-sparse==latest+cu101 \
  torch-scatter==latest+cu101 \
  torch-cluster==latest+cu101 \
  -f https://pytorch-geometric.com/whl/torch-1.4.0.html


Instruction: conv2d(): argument 'input' (position 1) must be Tensor, not str in loop function

Input:
I am trying to write a loop function to extract features from my data and save it in the list.

And here is my code:

import pickle

output_features = []

with torch.no_grad():
    for (inputs, labels) in dataloaders.items():
        x=feature_extractor(inputs)
        output_features.append(x)
        output_features = torch.cat(output_features).numpy

pickle.dump(output_features, open("features.pkl", "w"))

But i have an error.

conv2d(): argument 'input' (position 1) must be Tensor, not str

Then I changed again it to

with torch.no_grad():
    for i, (inputs, labels) in enumerate(dataloaders['val']):
        x=feature_extractor(inputs)
        output_features.append(x)
        output_features = torch.cat(output_features).numpy

pickle.dump(output_features, open("features.pkl", "wb"))

Then I got this error

'builtin_function_or_method' object has no attribute 'append'

Please, someone, explain to me why I got this error and how to get features from each input with its label and save them in an array. Thank you

Output:
For the first issue, it really depends on how the data is saved into the dataloader. But according to convention, I'm guessing it holds tuples of (input tensor, label), which can simply be accessed by for (inputs, labels) in dataloader:. The enumeration isn't compulsory, unless you would like to keep count of each iteration.

As for the second error, this is because you accidentally wrote output_features = torch.cat(output_features).numpy, which calls a built-in method, instead of applying it to the output tensor. You can correct this to output_features = torch.cat(output_features).numpy() and you should be able to get the output features fine!


Instruction: pytorch RuntimeError: Expected object of scalar type Double but got scalar type Float

Input:
I am trying to implement a custom dataset for my neural network. But got this error when running the forward function. The code is as follows.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
import numpy as np

class ParamData(Dataset):
    def __init__(self,file_name):
        self.data = torch.Tensor(np.loadtxt(file_name,delimiter = ','))    #first place
    def __len__(self):
        return self.data.size()[0]
    def __getitem__(self,i):
        return self.data[i]

class Net(nn.Module):
    def __init__(self,in_size,out_size,layer_size=200):
        super(Net,self).__init__()
        self.layer = nn.Linear(in_size,layer_size)
        self.out_layer = nn.Linear(layer_size,out_size)

    def forward(self,x):
        x = F.relu(self.layer(x))
        x = self.out_layer(x)
        return x

datafile = 'data1.txt'

net = Net(100,1)
dataset = ParamData(datafile)
n_samples = len(dataset)

#dataset = torch.Tensor(dataset,dtype=torch.double)   #second place
#net.float()                                          #thrid place

net.forward(dataset[0])         #fourth place

In the file data1.txt is a csv formatted text file containing certain numbers, and each dataset[i] is a size 100 by 1 torch.Tensor object of dtype torch.float64. The error message is as follows:

Traceback (most recent call last):
  File "Z:\Wrong.py", line 33, in <module>
    net.forward(dataset[0])
  File "Z:\Wrong.py", line 23, in forward
    x = F.relu(self.layer(x))
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\Python38\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "E:\Python38\lib\site-packages\torch\nn\functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm

It seems that I should change the dtype of the numbers in dataset to torch.double. I tried things like

changing the line at the first place to self.data = torch.tensor(np.loadtxt(file_name,delimiter = ','),dtype=torch.double)
changing the line at the fourth place to net.forward(dataset[0].double())
uncommenting one of the two lines at the second or the thrid place

I think these are the solutions I have seen from similar questions, but they either give new errors or don't do anything. What should I do?

Update: So I got it working by changing the first place to

self.data = torch.from_numpy(np.loadtxt(file_name,delimiter = ',')).float()

which is weird because it is exactly the opposite of the error message. Is this a bug? I'd still like some explaining.

Output:
Now that I have more experience with pytorch, I think I can explain the error message. It seems that the line

RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm

is actually refering to the weights of the linear layer when the matrix multiplication is called. Since the input is double while the weights are float, it makes sense for the line

output = input.matmul(weight.t())

to expect the weights to be double.


Instruction: PyTorch MaxPool2D unexpected behavior with padding=1

Input:
I was playing around with MaxPool2D in PyTorch and discovered strange behavior when setting padding=1. Here is what I got:

Code:

import torch
from torch.nn.functional import max_pool2d

TEST = 1

def test_maxpool(negative=False, tnsr_size=2, kernel_size=2, stride=2, padding=0):
    """Test MaxPool2D.
    """
    global TEST
    print(f'=== TEST {TEST} ===')
    print(*[f'{i[0]}: {i[1]}' for i in locals().items()], sep=' | ')

    inp = torch.arange(1., tnsr_size ** 2 + 1).reshape(1, tnsr_size, tnsr_size)
    inp = -inp if negative else inp

    print('In:')
    print(inp)

    out = max_pool2d(inp, kernel_size, stride, padding=padding)
    print('Out:')
    print(out)
    print()
    TEST += 1

test_maxpool()
test_maxpool(True)

test_maxpool(padding=1)
test_maxpool(True, padding=1)

Out:

=== TEST 1 ===
negative: False | tnsr_size: 2 | kernel_size: 2 | stride: 2 | padding: 0
In:
tensor([[[1., 2.],
         [3., 4.]]])
Out:
tensor([[[4.]]])

=== TEST 2 ===
negative: True | tnsr_size: 2 | kernel_size: 2 | stride: 2 | padding: 0
In:
tensor([[[-1., -2.],
         [-3., -4.]]])
Out:
tensor([[[-1.]]])

=== TEST 3 ===
negative: False | tnsr_size: 2 | kernel_size: 2 | stride: 2 | padding: 1
In:
tensor([[[1., 2.],
         [3., 4.]]])
Out:
tensor([[[1., 2.],
         [3., 4.]]])

=== TEST 4 ===
negative: True | tnsr_size: 2 | kernel_size: 2 | stride: 2 | padding: 1
In:
tensor([[[-1., -2.],
         [-3., -4.]]])
Out:
tensor([[[-1., -2.],
         [-3., -4.]]])

Tests 1, 2, 3 are fine but Test 4 is weird, I expected to get [[0 0], [0 0]] tensor:

In:
[[-1 -2]
 [-3 -4]]

+ padding ->

[[ 0  0  0  0]
 [ 0 -1 -2  0]
 [ 0 -3 -4  0]
 [ 0  0  0  0]]

-> kernel_size=2, stride=2 ->

[[0 0]
 [0 0]]

According to Test 3 zero padding was used but Test 4 produced controversial result.

What kind of padding (if any) was that? Why does MaxPool2D behave like that?

pytorch 1.3.1

Output:
This was expected behavior since negative infinity padding is done by default.
The documentation for MaxPool is now fixed. See this PR: Fix MaxPool default pad documentation #59404 .


Instruction: What is the meaning of the second output of Huggingface's Bert?

Input:
Using the vanilla configuration of base BERT model in the huggingface implementation, I get a tuple of length 2.

import torch

import transformers
from transformers import AutoModel,AutoTokenizer

bert_name="bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(bert_name)
BERT = AutoModel.from_pretrained(bert_name)

e=tokenizer.encode('I am hoping for the best', add_special_tokens=True)

q=BERT(torch.tensor([e]))

print (len(q)) #Output: 2

The first element is what I expect to receive - the 768 dimension embedding of each input token.

print (e) #Output : [101, 1045, 2572, 5327, 2005, 1996, 2190, 102]
print (q[0].shape) #Output : torch.Size([1, 8, 768])

But what is the second element in the tuple?

print (q[1].shape) # torch.Size([1, 768])

It has the same size as the encoding of each token.
But what is it?

Maybe a copy of the [CLS] token, a representation for the classification of the entire encoded text?

Let's check.

a= q[0][:,0,:]
b=q[1]

print (torch.eq(a,b)) #Output : Tensor([[False, False, False, .... False]])

Nope!

What about a copy the embedding of the last token (for whatever reason)?

c= q[0][:,-1,:]
b=q[1]

print (torch.eq(a,c)) #Output : Tensor([[False, False, False, .... False]])

So, also not that.

The documentation talks about how changing the config can result in more tuple elements (like hidden states), but I did not find any description of this "mysterious" tuple element outputted by the default configuration.

Any ideas as to what is it and what is its usage?

Output: The output in this case is a tuple of (last_hidden_state, pooler_output). You can find documentation about what the returns could be here.


Instruction: How necessary are activation functions after dense layer in neural networks?

Input:
I'm currently training multiple recurrent convolutional neural networks with deep q-learning for the first time.

Input is a 11x11x1 matrix, each network consists of 4 convolutional layer with dimensions 3x3x16, 3x3x32, 3x3x64, 3x3x64. I use stride=1 and padding=1. Each convLayer is followed by ReLU activation. The output is fed into a feedforward fully-connected dense layer with 128 units and after that into an LSTM layer, also containing 128 units. Two following dense layer produce separate advantage and value steams.

So training is running for a couple of days now and now I've realized (after I've read some related paper), I didn't add an activation function after the first dense layer (as in most of the papers). I wonder if adding one would significantly improve my network? Since I'm training the networks for university, I don't have unlimited time for training, because of a deadline for my work. However, I don't have enough experience in training neural networks, to decide on what to do...
What do you suggest? I'm thankful for every answer!

Output:
If I have to talk in general using an activation function helps you to include some non-linear property in your network.

The purpose of an activation function is to add some kind of non-linear property to the function, which is a neural network. Without the activation functions, the neural network could perform only linear mappings from inputs x to the outputs y. Why is this so?

Without the activation functions, the only mathematical operation during the forward propagation would be dot-products between an input vector and a weight matrix. Since a single dot product is a linear operation, successive dot products would be nothing more than multiple linear operations repeated one after the other. And successive linear operations can be considered as a one single learn operation.

A neural network without any activation function would not be able to realize such complex mappings mathematically and would not be able to solve tasks we want the network to solve.


Instruction: Read multiple .gz file and return it in one tensor

Input:
I am trying to read multiple .gz file and return its content in one tensor as follows:
with ReadHelper('ark: gunzip -c /home/mnabih/kaldi/egs/timit/s5/exp/mono_ali/*.gz|') as reader:
    for key, b in reader:
        #print(type(b))
        c = torch.from_numpy(b)
        labels = torch.cat(c)

Unfortunately, it gives me this error:

cat(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor

Output:
As the error message explains, c is a tensor. To use torch.cat() you must pass a group of tensors or a list. To solve your problem you may use:
temp = list()
for key, b in reader:
    temp.append(torch.from_numpy(b))
labels = torch.cat(temp)

For more, you can check the manual here


Instruction: How to fix multiprocessing problems in python in windows10

Input:
I try to use this tutorial to train my own car model recognition model: https://github.com/Helias/Car-Model-Recognition. And i want to use coda and my gpu perfomance to enhance training speed (preprocesssing step was completed without any errors).But when I try to train my model, I've got the following errors:

######### ERROR #######

        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

######### batch #######
Traceback (most recent call last):
  File "D:\Car-Model-Recognition\main.py", line 78, in train_model

######### ERROR #######
[Errno 32] Broken pipe
    for i, batch in enumerate(loaders[mode]):

######### batch #######  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__

    return _MultiProcessingDataLoaderIter(self)
Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
  File "main.py", line 78, in train_model
    w.start()
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    for i, batch in enumerate(loaders[mode]):
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    w.start()
    return Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    _check_not_importing_main()
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
    return Popen(process_obj)

I have used the exact code from given link, and if i start my code using wsl, everything is ok, but I can't use my gpu from wsl. Where should I insert this name == 'main' check to prevent such a mistake or how can i disable this multiprocessing

Output:
Looking at main.py, you run a lot of code at the module level. On Windows, python's multiprocessing module will start a new python interpreter, import your modules, unpickle a snapshot of your parent context and then call your worker function. The problem is that all of that module level code executes merely by import and you essentially run a new copy of your program instead of building a context for your worker.

The solution is two-fold. First, move all of the module level code into functions. You want to be a able to import your module without side effects. Second, call the function(s) that start your program from a conditional

def main():
    the stuff you were doing a module level

if __name__ == "__main__":
    main()

The reason this works is in the module name. When you run the top level script of a python (e.g., python main.py), its a script called "__main__", not a module. If a different program imports main its a module called "main" (or whatever you named your script). That 'if' stops your main code from executing if its imported by some other python code - such as the multiprocessing module.

Its okay to have some executable code at the module level, especially if you are setting up defaults and such. But don't do anything at the module level that you wouldn't want done if some other code imports your script.