Instruction: Dockerfile: pip install fails with requirements.txt (but succeeds with individual packages)

Input:
I'm trying to install some packages in a docker container, and there is a problem when installing from a requirements.txt file. This line:
RUN python3.8 -m pip install -r requirements.txt

fails with the error:
...
Collecting torch
  Downloading torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl (735.5 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    torch from https://files.pythonhosted.org/packages/89/c1/72e9050d3e31e4df983f6e06799a1a4c896427c1e5645a6d810940944b60/torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl#sha256=fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787 (from -r requirements.txt (line 3)):
        Expected sha256 fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787
             Got        d5466637c17c3ae0c81c00d93a0b7c8d8428cfd216f54953a11d0788ea7b74fb

The requirements.txt file is the following:
numpy
opencv-python
torch

However, when installing these packages one at a time everything works fine:
RUN python3.8 -m pip install numpy
RUN python3.8 -m pip install opencv-python
RUN python3.8 -m pip install torch

Any ideas how to solve this?
*** EDIT ***
Dockerfile up to that point:
FROM public.ecr.aws/lambda/python:3.8
COPY requirements.txt ./

Output:
You could try a couple of things. Depending on your base image, you could run pip install in this way:
RUN pip install -r requirements.txt

Another option would be to change your requirements.txt such that it is version controlled. Then you can be sure you have compatible versions and is a good practice in general. Eg.:
torch==1.8.0

Try to run Docker again with without caches:
docker build -no-cache

Or you could check this answer:
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. when updating Django


Instruction: ImportError: cannot import name 'download_url_to_file'

Input:
I tried to run a Python script that uses the download_url_to_file method inside the torch hub, but I got the following error:
Traceback (most recent call last):
  File "crop-video.py", line 1, in <module>
    import face_alignment
  File "$PROGRAMS_PATH$\Python\Python36\lib\site-packages\face_alignment-1.3.3-py3.6.egg\face_alignment\__init__.py", line 7, in <module>
  File "$PROGRAMS_PATH$\Python\Python36\lib\site-packages\face_alignment-1.3.3-py3.6.egg\face_alignment\api.py", line 7, in <module>
  File "$PROGRAMS_PATH$\Python\Python36\lib\site-packages\face_alignment-1.3.3-py3.6.egg\face_alignment\utils.py", line 13, in <module>
ImportError: cannot import name 'download_url_to_file'

I am using Python version 3.6.2. I also have the packages installed with the following versions:
certifi (2020.12.5)
chardet (4.0.0)
cycler (0.10.0)
dataclasses (0.8)
decorator (4.4.2)
face-alignment (1.3.3)
future (0.18.2)
idna (2.10)
imageio (2.9.0)
imageio-ffmpeg (0.4.3)
kiwisolver (1.3.1)
llvmlite (0.36.0)
matplotlib (3.3.4)
networkx (2.5)
numba (0.53.0)
numpy (1.19.5)
opencv-python (4.5.1.48)
Pillow (8.1.2)
pip (9.0.1)
pyparsing (2.4.7)
python-dateutil (2.8.1)
PyWavelets (1.1.1)
requests (2.25.1)
rocketchat-API (0.6.9)
scikit-image (0.17.2)
scipy (1.5.4)
setuptools (28.8.0)
six (1.15.0)
tifffile (2020.9.3)
torch (1.0.0)
torchvision (0.2.1)
tqdm (4.24.0)
typing-extensions (3.7.4.3)
unknown (0.0.0)
urllib3 (1.26.4)

The download_url_to_file method does not appear to be found inside the hub file, but I checked the torch hub, and made sure it was defined! How can this error be fixed?

Output:
Worked in my Pc
use torch == 1.6


Instruction: PyTorch gather 3D source with 2D index

Input:
Given a source tensor of shape [B,N,F] and an index tensor of shape [B,k], where index[i][j] is an index to a specific feature inside source[i][j] is there a way to extract an output tensor such that:
output[i][j] = source[i][j][index[i][j]]

torch.gather specifies that index.shape == source.shape, while here the shape of the source is one dimension bigger.
source = [
[[0.1,0.2],[0.2,0.3]],
[[0.4,0.5],[0.6,0.7]],
[[0.7,0.6],[0.8,0.9]]
]
index = [
[1,0],
[0,0],
[1,1]
]

desired_output = [
[0.2,0.2],
[0.4,0.6],
[0.6,0.9]
]

Output:
For future references - The solution is
source.gather(2,index.unsqueeze(2)).squeeze(2)


Instruction: Tokenizing & encoding dataset uses too much RAM

Input:
Trying to tokenize and encode data to feed to a neural network.
I only have 25GB RAM and everytime I try to run the code below my google colab crashes. Any idea how to prevent his from happening? “Your session crashed after using all available RAM”
I thought tokenize/encoding chunks of 50000 sentences would work but unfortunately not.
The code works on a dataset with length 1.3 million. The current dataset has a length of  5 million.
max_q_len = 128
max_a_len = 64
trainq_list = train_q.tolist()
batch_size = 50000

def batch_encode(text, max_seq_len):
      for i in range(0, len(trainq_list), batch_size):
        encoded_sent = tokenizer.batch_encode_plus(
            text,
            max_length = max_seq_len,
            pad_to_max_length=True,
            truncation=True,
            return_token_type_ids=False
        )
      return encoded_sent

    # tokenize and encode sequences in the training set
    tokensq_train = batch_encode(trainq_list, max_q_len)

The tokenizer comes from HuggingFace:
tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-uncased')

Output:
You should use generators and pass data to tokenizer.batch_encode_plus, no matter the size.
Conceptually, something like this:
Training list
This one probably holds list of sentences, which is read from some file(s). If this is a single large file, you could follow this answer to lazily read parts of the input (preferably of batch_size lines at once):
def read_in_chunks(file_object, chunk_size=1024):
    """Lazy function (generator) to read a file piece by piece.
    Default chunk size: 1k."""
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data

Otherwise open a single file (much smaller than memory, because it will be way larger after encoding using BERT), something like this:
import pathlib

def read_in_chunks(directory: pathlib.Path):
    # Use "*.txt" or any other extension your file might have
    for file in directory.glob("*"):
        with open(file, "r") as f:
            yield f.readlines()

Encoding
Encoder should take this generator and yield back encoded parts, something like this:
# Generator should create lists useful for encoding
def batch_encode(generator, max_seq_len):
    tokenizer = BertTokenizerFast.from_pretrained("bert-base-multilingual-uncased")
    for text in generator:
        yield tokenizer.batch_encode_plus(
            text,
            max_length=max_seq_len,
            pad_to_max_length=True,
            truncation=True,
            return_token_type_ids=False,
        )

Saving encoded files
As the files will be too large to fit in RAM memory, you should save them to disk (or use somehow as they are generated).
Something along those lines:
import numpy as np

# I assume np.arrays are created, adjust to PyTorch Tensors or anything if needed
def save(encoding_generator):
    for i, encoded in enumerate(encoding_generator):
        np.save(str(i), encoded)


Instruction: Can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

Input:
I am trying to show results of GAN network on some specified epochs. The function for printing the current result was used previously with TF. I need to change in to pytorch.
def show_result(G_net, z_, num_epoch, show=False, save=False, path='result.png'):

  #test_images = sess.run(G_z, {z: z_, drop_out: 0.0})
  test_images = G_net(z_)

  size_figure_grid = 5
  fig, ax = plt.subplots(size_figure_grid, size_figure_grid, figsize=(5, 5))

  for i, j in itertools.product(range(size_figure_grid), range(size_figure_grid)):
     ax[i, j].get_xaxis().set_visible(False)
     ax[i, j].get_yaxis().set_visible(False)

  for k in range(5*5):
     i = k // 5
     j = k % 5
     ax[i, j].cla()
     ax[i, j].imshow(np.reshape(test_images[k], (28, 28)), cmap='gray')

  label = 'Epoch {0}'.format(num_epoch)
  fig.text(0.5, 0.04, label, ha='center')

  plt.savefig(name)
  file = drive.CreateFile({'title': label, "parents": [{"kind": "https://drive.google.com/drive/u/0/folders/", "id": folder_id}]})
  file.SetContentFile(name)
  file.Upload()

  if num_epoch == 10 or num_epoch == 20 or num_epoch == 50 or num_epoch == 100:
     plt.show()

  plt.close()

The results I need to obtain looks like that:
result img
I am getting this error, however I am not sure what I did incorrectly

Can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

Output:
I am assuming G_net is your network. It looks like you store the network in the GPU, hence the returned result test_images will also be in GPU. You will need to move it to cpu and convert to numpy:
#test_images = G_net(z_)
test_images = G_net(z_).detach().cpu().numpy()

This will detach the tensor from the graph, move to cpu and then convert to numpy.


Instruction: CUDA out of memory.Tried to allocate 14.00 MiB (GPU 0;4.00 GiB total capacity;2 GiB already allocated;6.20 MiB free;2GiB reserved intotal by PyTorch)

Input:
I am trying to run this code from fastai
from fastai.vision.all import *
path = untar_data(URLs.PETS)/'images'

def is_cat(x): return x[0].isupper()
dls = ImageDataLoaders.from_name_func(
    path, get_image_files(path), valid_pct=0.2, seed=42,
    label_func=is_cat, item_tfms=Resize(224), num_workers = 0)

learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(1)

I get the following error

RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0;
4.00 GiB total capacity; 2.20 GiB already allocated; 6.20 MiB free; 2.23 GiB reserved in total by PyTorch)

I also tried running
import torch
torch.cuda.empty_cache()

and restarting the kernel which was of no use
Any help would be appreciated

Output: The default batch_size used in the ImageDataLoaders.from_name_func is 64 according to the documentation here. Reducing that should solve your problem. Pass another parameter to ImageDataLoaders.from_name_func like bs=32 or any other smaller value till the error is not thrown


Instruction: RuntimeError:shape ‘[4, 98304]’ is invalid for input of size 113216

Input:
I am learning to train a basic nn model for image classification, the error happened when I was trying to feed in image data into the model. I understand that I should input correct size of image data. My image data is 128*256 with 3 channels,4 classes, and the batch size is 4. What I don't understand is where does the size 113216 come from? I checked all related parameters or image meta data, but didn't find a clue. Here is my code:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(3*128*256, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(4, 3*128*256)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
for epoch in range(2):  # loop over the dataset multiple times
    print('round start')
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        print(inputs.shape)
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

Thanks for your help!

Output:
Shapes

Conv2d changes width and height of image without padding. Rule of thumb (if you want to keep the same image size with stride=1 (default)): padding = kernel_size // 2
You are changing number of channels, while your linear layer has 3 for some reason?
Use print(x.shape) after each step if you want to know how your tensor data is transformed!

Commented code
Fixed code with comments about shapes after each step:
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = torch.nn.Conv2d(3, 6, 5)
        self.pool = torch.nn.MaxPool2d(2, 2)
        self.conv2 = torch.nn.Conv2d(6, 16, 5)
        # Output shape from convolution is input shape to fc
        self.fc1 = torch.nn.Linear(16 * 29 * 61, 120)
        self.fc2 = torch.nn.Linear(120, 84)
        self.fc3 = torch.nn.Linear(84, 10)

    def forward(self, x):
        # In: (4, 3, 128, 256)
        x = F.relu(self.conv1(x))
        # (4, 3, 124, 252) because kernel_size=5 takes 2 pixels
        x = self.pool(x)
        # (4, 6, 62, 126) # Because pooling halving the size
        x = F.relu(self.conv2(x))
        # (4, 16, 58, 122) # Same reason as above
        x = self.pool(x)
        # (4, 16, 29, 61) Because pooling halving the size
        # Better use torch.flatten(x, dim=1) so you don't have to input size here
        x = x.view(-1, 16 * 29 * 61)  # Use -1 to be batch size independent
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Other things that might help

Try torch.nn.AdaptiveMaxPool2d(1) before ReLU, it will make your network width and height independent
Use flatten (or torch.nn.Flatten() layer) after this pooling
If so, pass num_channels set in last convolution as in_features for nn.Linear


Instruction: Pytorch random tensor generation with one fix value and rest of random value

Input:
In my testing dataset, I have to always include one specific image(image at position 0) in each batch but others values can be randomly selected. So I am making a tensor which will have 1st value 0 (for 1st image) and the rest of others can be anything other than 0. My code snippet is below.
a= torch.randperm(len(l-1)) #where l is total no of testing image in dataset, code output->tensor([10, 0, 1, 2, 4, 5])
b=torch.tensor([0]) # code output-> tensor([0])
c=torch.cat((b.view(1),a))# gives output as -> tensor([0, 10, 0, 1, 2, 4, 5]) and 0 is used twice so repeated test image

However, above approach can include 0 twice as torch.randperm many times includes 0. Is there a way in torch to generate random number skipping one specific value. Or if you think another approach will be better please comment.

Output:
You could just remove these 0s using conditional indexing (also assumed you meant len(l) - 1):
a= torch.randperm(len(l)-1) #where l is total no of testing image in dataset, code output->tensor([10, 0, 1, 2, 4, 5])
a=a[a!=0]
b=torch.tensor([0]) # code output-> tensor([0])
c=torch.cat((b,a))# gives output as -> tensor([0, 10, 0, 1, 2, 4, 5]) and 0 is used twice so repeated test image

Or if you want to make sure it's never put in:
a=torch.arange(1,len(l))
a=a[torch.randperm(a.shape[0])]
b=torch.tensor([0])
c=torch.cat((b,a))

The second approach is a bit more versatile as you can have whatever values you'd like in your initial a declaration as well as replacement.


Instruction: How to replace specific values in PyTorch tensor along diagonal?

Input:
For example, there is a PyTorch matrix A:
A = tensor([[3,2,1],[1,0,2],[2,2,0]])

I need to replace 0 with 1 on the diagonal, so the result should be:
tensor([[3,2,1],[1,1,2],[2,2,1]])

Output:
You can use torch's inbuilt diagonal functions to replace diagonal elements like so:
mask = A.diagonal() == 0
A += torch.diag(mask)

>>> A
tensor([[3, 2, 1],
        [1, 1, 2],
        [2, 2, 1]])

If you want to replace 0's with another value, change mask to mask * replace_value.


Instruction: Confusion about Pytorch `torch.split` documentation

Input:
When I view the explanation of the function torch.split in PyTorch, I find it difficult for me to read as a non-English-speaker:

torch.split(tensor, split_size_or_sections, dim=0)
[...]
If split_size_or_sections is a list, then tensor will be split
into len(split_size_or_sections) chunks with sizes in dim according
to split_size_or_sections.

Does "with sizes in dim" mean "with sizes in split_size_or_sections along the dimension dim"?

Output:
Don't worry - your English is fine, that line is a bit confusing.
Yes you're correct. It means if you pass a list e.g. split_size_or_sections=[1,2,4,5] it will split the tensor into len([1,2,4,5]) chunks (with the splits happening across dim), and each chunk will be of length 1, 2, 4, 5 respectively.
This implicitly assumes that sum([1,2,4,5]) equals the size of dim, and will return an error if not.


Instruction: Two tensors are on the same device, but I get the error: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu

Input:
I have:
def loss_fn(self, pred, truth):
    truth_flat = torch.reshape(truth, (truth.size(0),-1)).to(truth.device)
    pred_flat = torch.reshape(pred, (pred.size(0),-1)).to(pred.device)

    stoi_loss = NegSTOILoss(sample_rate=16000)(pred_flat, truth_flat)
    print('truth', truth.size(), truth_flat.size(), stoi_loss)

    return torch.nn.MSELoss()(pred, truth)

As you can see, I'm making sure that it's on the same device, but I still get the error:
Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu

Any ideas?

Output: You are assigning to two different devices, truth.device and pred.device.


Instruction: How can I set the diagonal of an N-dim tensor to 0 along given dims?

Input:
I’m trying to figure out a way to set the diagonal of a 3-dimensional Tensor (along 2 given dims) equal to 0. An example of this would be, let’s say I have a Tensor of shape [N,N,N] and I wanted to set the diagonal along dim=1,2 equal to 0? How exactly could that be done?
I tried using fill_diagonal_ but that only does the k-th diagonal element for each sub-array, i.e:
>>> data = torch.ones(3,4,4)
>>> data.fill_diagonal_(0)

tensor([[[0, 1, 1, 1],
         [1, 1, 1, 1],
         [1, 1, 1, 1],
         [1, 1, 1, 1]],

        [[1, 1, 1, 1],
         [1, 0, 1, 1],
         [1, 1, 1, 1],
         [1, 1, 1, 1]],

        [[1, 1, 1, 1],
         [1, 1, 1, 1],
         [1, 1, 0, 1],
         [1, 1, 1, 1]]])

whereas I would want the entire diagonal for each sub-matrix to be equal to 0 here. So, the desired outcome would be,
tensor([[[0, 1, 1, 1],
         [1, 0, 1, 1],
         [1, 1, 0, 1],
         [1, 1, 1, 0]],

        [[0, 1, 1, 1],
         [1, 0, 1, 1],
         [1, 1, 0, 1],
         [1, 1, 1, 0]],

        [[0, 1, 1, 1],
         [1, 0, 1, 1],
         [1, 1, 0, 1],
         [1, 1, 1, 0]]])

Secondly, the reason I state for a given pair of dimension is, I need to repeat this `zeroing’ along 2 different pairs of dimensions (e.g. dim=(1,2) then dim=(0,1)) to get the required masking I need.
Is there a way to mask a given diagonal over 2 arbitrary dimensions for a 3D-tensor?

Output:
You can do this with a for loop over the sub-tensors:
# across dim0
for i in range(data.size(0)):
    data[i].fill_diagonal_(0)

If you need to perform this over an arbitrary two dimensions of a 3d tensor, simply apply the fill to the appropriate slices:
# across dim1
for i in range(data.size(1)):
    data[:,i].fill_diagonal_(0)

# across dim2
for i in range(data.size(2)):
    data[:,:,i].fill_diagonal_(0)


Instruction: How to convert a full string output into torch tensor

Input:
Having defined an API based on a transformer model which I send request to, I got as output an object of class string supposed to represent a torch tensor.
import requests
import json
import torch

response = requests.post("http://127.0.0.1:5000/predict", json="This is a test.")
data = response.json()
print(data)

tensor([[-5.5169e-01,  5.1988e-01, -1.4731e-01,  4.6096e-02, -5.5753e-02,
         -6.9530e-01,  7.3424e-01,  3.0014e-01,  4.0528e-01, -1.7587e-01,
          1.2586e-01,  5.6712e-01, -4.0757e-01,  1.5796e-01,  9.4700e-01,
          6.2967e-01,  3.1027e-01,  1.0169e-02,  3.1380e-02,  1.2585e-01,
          2.3633e-01,  6.0813e-01, -1.0548e+00, -1.8581e-01, -5.9870e-02,
          1.2952e-01, -3.8818e-01,  2.3425e-01]])

I checked in the requests module documentation, but couldn't find any way to convert this fully strings into a torch tensor?

Output:
You can use eval to evaluate the code contained in the string:
from torch import tensor

my_tensor = eval(data)


Instruction: How to return a k-dim pytorch tensor truncated based on 1D mask

Input:
So suppose I have a k-dim tensor and a 1-dim mask, that is used a lot in pytorch for variable length sequences, and I want to return a tensor that represents the elements up to the first false value in mask.  Here is an example:
import torch

a = torch.tensor([[1,2],[3,4],[5,6],[0,0],[0,0],[0,0]])
b = torch.tensor([True,True,True,False,False,False])

# magic goes here, result of c should be:

print(c)
>>> [[1,2],[3,4],[5,6]]

In this example, the input tensor is 2D, but it could be k-d, with any number of values across those dimensions.  Only the first dimension needs to match the mask dimension.  So doing torch.masked_select doesn't work since the tensor to be truncated isn't 1D like the mask, and since you don't know the dimensionality, squeeze and unsqueeze is not a solution either.
The mask is always going to be true for the first k elements and false for the remaining elements, though if your solution does not "depend" on this, that is fine.
This seems like people would be doing it all the time, yet I cannot find anywhere where this question has been answered.

Output:
You can simply pass the mask as a slice index to the tensor:
c = a[b]

>>> c
tensor([[1, 2],
        [3, 4],
        [5, 6]])


Instruction: Rearrange torch 2D tensors ("Tiles") to be in a particular order

Input:
I have something that looks like this...
import torch
X = torch.cat([torch.ones((2,4)).unsqueeze(0), torch.ones((2,4)).unsqueeze(0)*2, torch.ones((2,4)).unsqueeze(0)*3, torch.ones((2,4)).unsqueeze(0)*4]).unsqueeze(0)

which is
tensor([[[[1., 1., 1., 1.],
          [1., 1., 1., 1.]],

         [[2., 2., 2., 2.],
          [2., 2., 2., 2.]],

         [[3., 3., 3., 3.],
          [3., 3., 3., 3.]],

         [[4., 4., 4., 4.],
          [4., 4., 4., 4.]]]])

And I want to rearrange them to look like this
tensor([[[1., 1., 1., 1., 2., 2., 2., 2.],
         [1., 1., 1., 1., 2., 2., 2., 2.],
         [3., 3., 3., 3., 4., 4., 4., 4.],
         [3., 3., 3., 3., 4., 4., 4., 4.]]])

However the solution I was hoping would work obviously doesn't and while I understand why it doesn't work I'll share it.
b, t, x, y = X.shape
num_x_splits = num_y_splits = 2
X = X.reshape(-1,x*num_x_splits,y*num_y_splits)

tensor([[[1., 1., 1., 1., 1., 1., 1., 1.],
         [2., 2., 2., 2., 2., 2., 2., 2.],
         [3., 3., 3., 3., 3., 3., 3., 3.],
         [4., 4., 4., 4., 4., 4., 4., 4.]]])

Is there a computationally efficient way I can accomplish what I want?

I did some more research and looks like the ideal solution might actually be a combinatoin of the accepted answer and to use the einops package which provides a high-level, easy-to-read, efficient solution

Output:
I will start with the simple case where we know the number of tensors and which tensors to be positioned etc.:
ones = 1 * torch.ones(2,4)
twos = 2 * torch.ones(2,4)
thrs = 3 * torch.ones(2,4)
fors = 4 * torch.ones(2,4)
fivs = 5 * torch.ones(2,4)
sixs = 6 * torch.ones(2,4)
row1 = torch.cat([ones, twos], axis = 1)
row2 = torch.cat([thrs, fors], axis = 1)
row3 = torch.cat([fivs, sixs], axis = 1)
comb = torch.cat([row1, row2, row3], axis = 0)
print(comb)

tensor([[1., 1., 1., 1., 2., 2., 2., 2.],
        [1., 1., 1., 1., 2., 2., 2., 2.],
        [3., 3., 3., 3., 4., 4., 4., 4.],
        [3., 3., 3., 3., 4., 4., 4., 4.],
        [5., 5., 5., 5., 6., 6., 6., 6.],
        [5., 5., 5., 5., 6., 6., 6., 6.]])

So you just need to concatenate rows first then combine all rows on the axis=0 which will get the positioning as you want if you know which tensors you want to combine etc.
Another way to solve this is to permute the dimensions so that you get the placement you want which should work for more complex cases as well:
comb2 = torch.cat([ones, twos, thrs, fors, fivs, sixs]).view(3, 2, 2, 4)
print(comb2)
print(comb2.permute(0, 2, 1, 3). reshape(6, 8))

tensor([[[[1., 1., 1., 1.],
          [1., 1., 1., 1.]],

         [[2., 2., 2., 2.],
          [2., 2., 2., 2.]]],

        [[[3., 3., 3., 3.],
          [3., 3., 3., 3.]],

         [[4., 4., 4., 4.],
          [4., 4., 4., 4.]]],

        [[[5., 5., 5., 5.],
          [5., 5., 5., 5.]],

         [[6., 6., 6., 6.],
          [6., 6., 6., 6.]]]])
tensor([[1., 1., 1., 1., 2., 2., 2., 2.],
        [1., 1., 1., 1., 2., 2., 2., 2.],
        [3., 3., 3., 3., 4., 4., 4., 4.],
        [3., 3., 3., 3., 4., 4., 4., 4.],
        [5., 5., 5., 5., 6., 6., 6., 6.],
        [5., 5., 5., 5., 6., 6., 6., 6.]])

In the first view of the tensor, we view it such that we know we have 3 sets of tensors with 2x4 dimensions and each of these 2x4 tensors stored in additional dimension (2x2x4) to make it easier to manipulate instead of storing 2x8 or 4x4. Then we rearrange the dimensions to get the placement you want. I try to go backwards from the desired dimension to find the permutation I want usually. First dimension should then be 6 = 3x2 where we get 3 sets and 2 rows of tensor so we keep the first axis in place, move rows dimension next to set dimension: permute(0, 2, 1, 3). Permutation of 4 dimensional tensors can be a bit tricky to imagine, just play with it and you will get the right combination :)


Instruction: Wall damage detection

Input:
I'm lanning to create a real-time wall damages detector [scratches, Cracks] using YOLOv5 and my custom dataset of images (125).

Do you think I can do transfer learning or it won’t be possible since the coco dataset classes are not similar?
Do you think I need to increase the dataset size?

For now I'm just trying to do a proof of concept. Wanted to have my steps planned in advance.

Output: If your dataset is small (like in you case), transfer learning will almost always give better results when compared to training from scratch. As for your second question, yes. The more data you get, the better your model will be able to learn and perform. Considering that it's a relatively different task than which Yolo-V5 was originally trained on, try to get as many images as you can


Instruction: Training accuracy decrease and loss increase when using pack_padded_sequence - pad_packed_sequence

Input:
I'm trying to train a bidirectional lstm with pack_padded_sequence and pad_packed_sequence, but the accuracy keeps decreasing while the loss increasing.
This is my data loader:
X1 (X[0]): tensor([[1408, 1413,   43,  ...,    0,    0,    0],
    [1452, 1415, 2443,  ...,    0,    0,    0],
    [1434, 1432, 2012,  ...,    0,    0,    0],
    ...,
    [1408, 3593, 1431,  ...,    0,    0,    0],
    [1408, 1413, 1402,  ...,    0,    0,    0],
    [1420, 1474, 2645,  ...,    0,    0,    0]]), shape: torch.Size([64, 31])

len_X1 (X[3]): [9, 19, 12, 7, 7, 15, 4, 13, 9, 8, 14, 19, 7, 23, 7, 13, 7, 12, 10, 12, 13, 11, 31, 8, 20, 17, 8, 9, 9, 29, 8, 5, 5, 13, 9, 8, 10, 17, 13, 8, 8, 11, 7, 29, 15, 10, 6, 7, 10, 9, 10, 10, 4, 16, 11, 10, 16, 8, 13, 8, 8, 20, 7, 12]

X2 (X[1]): tensor([[1420, 1415,   51,  ...,    0,    0,    0],
        [1452, 1415, 2376,  ..., 1523, 2770,   35],
        [1420, 1415,   51,  ...,    0,    0,    0],
        ...,
        [1408, 3593, 1474,  ...,    0,    0,    0],
        [1408, 1428, 2950,  ...,    0,    0,    0],
        [1474, 1402, 3464,  ...,    0,    0,    0]]), shape: torch.Size([64, 42])

len_X2 (X[4]): [14, 42, 13, 18, 12, 31, 8, 19, 5, 7, 15, 19, 7, 17, 6, 11, 12, 16, 8, 8, 19, 8, 12, 10, 11, 9, 9, 9, 9, 21, 7, 5, 8, 13, 14, 8, 15, 8, 8, 8, 12, 13, 7, 14, 4, 10, 6, 11, 12, 7, 8, 11, 9, 13, 30, 10, 15, 9, 9, 7, 9, 8, 7, 20]

t (X[2]): tensor([0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1,
        0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0,
        0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1]), shape: torch.Size([64])

This is my model class:
class BiLSTM(nn.Module):
def __init__(self, n_vocabs, embed_dims, n_lstm_units, n_lstm_layers, n_output_classes):
    super(BiLSTM, self).__init__()
    self.v = n_vocabs
    self.e = embed_dims
    self.u = n_lstm_units
    self.l = n_lstm_layers
    self.o = n_output_classes
    self.padd_idx = tokenizer.get_vocab()['[PAD]']
    self.embed = nn.Embedding(
        self.v,
        self.e,
        self.padd_idx
        )
    self.bilstm = nn.LSTM(
        self.e,
        self.u,
        self.l,
        batch_first = True,
        bidirectional = True,
        dropout = 0.5
        )
    self.linear = nn.Linear(
        self.u * 4,
        self.o
        )

  def forward(self, X):
    # initial_hidden
    h0 = torch.zeros(self.l * 2, X[0].size(0), self.u).to(device)
    c0 = torch.zeros(self.l * 2, X[0].size(0), self.u).to(device)

    # embedding
    out1 = self.embed(X[0].to(device))
    out2 = self.embed(X[1].to(device))

    # # pack_padded_sequence
    out1 = nn.utils.rnn.pack_padded_sequence(out1, X[3], batch_first=True, enforce_sorted=False)
    out2 = nn.utils.rnn.pack_padded_sequence(out2, X[4], batch_first=True, enforce_sorted=False)

    # NxTxh, lxNxh
    out1, _ = self.bilstm(out1, (h0, c0))
    out2, _ = self.bilstm(out2, (h0, c0))

    # # pad_packed_sequence
    out1, _ = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
    out2, _ = nn.utils.rnn.pad_packed_sequence(out2, batch_first=True)

    # take only the final time step
    out1 = out1[:, -1, :]
    out2 = out2[:, -1, :]

    # concatenate out1&2
    out = torch.cat((out1, out2), 1)

    # linear layer
    out = self.linear(out)

    iout = torch.max(out, 1)[1]
    return iout, out

And if I remove pack_padded_sequence - pad_packed_sequence, the model training works just fine:
class BiLSTM(nn.Module):
def __init__(self, n_vocabs, embed_dims, n_lstm_units, n_lstm_layers, n_output_classes):
    super(BiLSTM, self).__init__()
    self.v = n_vocabs
    self.e = embed_dims
    self.u = n_lstm_units
    self.l = n_lstm_layers
    self.o = n_output_classes
    self.padd_idx = tokenizer.get_vocab()['[PAD]']
    self.embed = nn.Embedding(
        self.v,
        self.e,
        self.padd_idx
        )
    self.bilstm = nn.LSTM(
        self.e,
        self.u,
        self.l,
        batch_first = True,
        bidirectional = True,
        dropout = 0.5
        )
    self.linear = nn.Linear(
        self.u * 4,
        self.o
        )

  def forward(self, X):
    # initial_hidden
    h0 = torch.zeros(self.l * 2, X[0].size(0), self.u).to(device)
    c0 = torch.zeros(self.l * 2, X[0].size(0), self.u).to(device)

    # embedding
    out1 = self.embed(X[0].to(device))
    out2 = self.embed(X[1].to(device))

    # pack_padded_sequence
    # out1 = nn.utils.rnn.pack_padded_sequence(out1, X[3], batch_first=True, enforce_sorted=False)
    # out2 = nn.utils.rnn.pack_padded_sequence(out2, X[4], batch_first=True, enforce_sorted=False)

    # NxTxh, lxNxh
    out1, _ = self.bilstm(out1, (h0, c0))
    out2, _ = self.bilstm(out2, (h0, c0))

    # pad_packed_sequence
    # out1, _ = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
    # out2, _ = nn.utils.rnn.pad_packed_sequence(out2, batch_first=True)

    # take only the final time step
    out1 = out1[:, -1, :]
    out2 = out2[:, -1, :]

    # concatenate out1&2
    out = torch.cat((out1, out2), 1)

    # linear layer
    out = self.linear(out)

    iout = torch.max(out, 1)[1]
    return iout, out

Output:
These lines of your code are wrong.
# take only the final time step
out1 = out1[:, -1, :]
out2 = out2[:, -1, :]

You say you are taking the final time step but you are forgetting that each sequence has different lengths.
nn.utils.rnn.pad_packed_sequence will pad the output of each sequence until it's length equals that of the longest so that they all have the same length.
In other words you are slicing out vectors of zeros (the padding) for most sequence.
This should do what you want.
# take only the final time step
out1 = out1[range(out1.shape[0]), X3 - 1, :]
out2 = out2[range(out2.shape[0]), X4 - 1, :]

This is assuming X3 and X4 are tensors.


Instruction: Check if each element of a tensor is contained in a list

Input:
Say I have a tensor A and a container of values vals. Is there a clean way of returning a Boolean tensor of the same shape as A with each element being whether that element of A is contained within vals? e.g:
A = torch.tensor([[1,2,3],
                  [4,5,6]])
vals = [1,5]

# Desired output
torch.tensor([[True,False,False],
              [False,True,False]])

Output:
You can simply do like this:
result = A.apply_(lambda x: x in vals).bool()

Then result will contain this tensor:
tensor([[ True, False, False],
        [False,  True, False]])

Here I simply used a lambda function and the apply_ method that you can find in the official documentation.


Instruction: What does the * sign mean in this NN built by Pytorch?

Input:
I was reading the code for Generative Adversarial Nets Code by https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/gan/gan.py, I would like to know what the * sign means here, I searched on Google and Stackoverflow but could not find a clear explanation.
class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()

        def block(in_feat, out_feat, normalize=True):
            layers = [nn.Linear(in_feat, out_feat)]
            if normalize:
                layers.append(nn.BatchNorm1d(out_feat, 0.8))
            layers.append(nn.LeakyReLU(0.2, inplace=True))
            return layers

        self.model = nn.Sequential(
            *block(opt.latent_dim, 128, normalize=False),
            *block(128, 256),
            *block(256, 512),
            *block(512, 1024),
            nn.Linear(1024, int(np.prod(img_shape))),
            nn.Tanh()
        )

Output:
*x is iterable unpacking notation in Python. See this related answer.
def block returns a list of layers, and *block(...) unpacks the returned list into positional arguments to the nn.Sequential call.
Here's a simpler example:
def block(in_feat, out_feat):
    return (nn.Linear(in_feat, out_feat), nn.LeakyReLU(0.2, inplace=True))

self.model = nn.Sequential(
    *block(128, 256),
)

# Equivalent to:
# layers = block(128, 256)
# self.model = nn.Sequential(layers[0], layers[1])

# Also equivalent to:
# layers = block(128, 256)
# self.model = nn.Sequential(*layers)


Instruction: torch : Dijkstra's algorithm

Input:
I am working on 3D point clouds. I have the SPARSE MATRIX representation of the graph structure of the point cloud (like csr_matrix in scipy.sparse). I want to club together the points that are within certain threshold of the Geodesic distance (approximated by the path length in the graph) and process them together. TO FIND such points, I need to run some shortest path finding algorithm like Dijkstra's. In a nutshell, my idea is like this

Sample K points out of N points (that I could do using Furthest Point Sampling)
Find the nearest Geodesic neighbours (using BackProp supported algorithm) for each of K points
Process the neighbours for each point using some Neural Network

This will go in my forward function.
Is there a way to implement Dijkstra’s in my functionality?
Or any other idea that I can implement?
Thank you very much!

Output:
I created my custom implementation for Dijkstra using priority queues as discussed here
For the same, I created a custom PriorityQ class using torch function as below
class priorityQ_torch(object):
    """Priority Q implelmentation in PyTorch

    Args:
        object ([torch.Tensor]): [The Queue to work on]
    """

    def __init__(self, val):
        self.q = torch.tensor([[val, 0]])
        # self.top = self.q[0]
        # self.isEmpty = self.q.shape[0] == 0

    def push(self, x):
        """Pushes x to q based on weightvalue in x. Maintains ascending order

        Args:
            q ([torch.Tensor]): [The tensor queue arranged in ascending order of weight value]
            x ([torch.Tensor]): [[index, weight] tensor to be inserted]

        Returns:
            [torch.Tensor]: [The queue tensor after correct insertion]
        """
        if type(x) == np.ndarray:
            x = torch.tensor(x)
        if self.isEmpty():
            self.q = x
            self.q = torch.unsqueeze(self.q, dim=0)
            return
        idx = torch.searchsorted(self.q.T[1], x[1])
        print(idx)
        self.q = torch.vstack([self.q[0:idx], x, self.q[idx:]]).contiguous()

    def top(self):
        """Returns the top element from the queue

        Returns:
            [torch.Tensor]: [top element]
        """
        return self.q[0]

    def pop(self):
        """pops(without return) the highest priority element with the minimum weight

        Args:
            q ([torch.Tensor]): [The tensor queue arranged in ascending order of weight value]

        Returns:
            [torch.Tensor]: [highest priority element]
        """
        if self.isEmpty():
            print("Can Not Pop")
        self.q = self.q[1:]

    def isEmpty(self):
        """Checks is the priority queue is empty

        Args:
            q ([torch.Tensor]): [The tensor queue arranged in ascending order of weight value]

        Returns:
            [Bool] : [Returns True is empty]
        """
        return self.q.shape[0] == 0

Now dijkstra, with adjacency matrix(with graph weights as input)
def dijkstra(adj):
    n = adj.shape[0]
    distance_matrix = torch.zeros([n, n])
    for i in range(n):
        u = torch.zeros(n, dtype=torch.bool)
        d = np.inf * torch.ones(n)
        d[i] = 0
        q = priorityQ_torch(i)
        while not q.isEmpty():
            v, d_v = q.top()  # point and distance
            v = v.int()
            q.pop()
            if d_v != d[v]:
                continue
            for j, py in enumerate(adj[v]):
                if py == 0 and j != v:
                    continue
                else:
                    to = j
                    weight = py
                    if d[v] + py < d[to]:
                        d[to] = d[v] + py
                        q.push(torch.Tensor([to, d[to]]))
        distance_matrix[i] = d
    return distance_matrix

Returns shortest path distance matrix for the graph points!


Instruction: Why a.storage() is b.storage() returns false when a and b reference the same data?

Input:
>>> a = torch.arange(12).reshape(2, 6)
>>> a
tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11]])
>>> b = a[1:, :]
>>> b.storage() is a.storage()
False

But
>>> b[0, 0] = 999
>>> b, a # both tensors are changed
(tensor([[999,   7,   8,   9,  10,  11]]),
 tensor([[  0,   1,   2,   3,   4,   5],
         [999,   7,   8,   9,  10,  11]]))

What is exactly the objects that stores tensor data? How can I make check if 2 tensors share memory?

Output:
torch.Tensor.storage() returns a new instance of torch.Storage on every invocation. You can see this in the following
a.storage() is a.storage()
# False

To compare the pointers to the underlying data, you can use the following:
a.storage().data_ptr() == b.storage().data_ptr()
# True

There is a discussion of how to determine whether pytorch tensors share memory in this pytorch forum post.

Note the difference between a.data_ptr() and a.storage().data_ptr(). The first returns the pointer to the first element of the tensor, whereas the second seems to the point to the memory address of the underlying data (not the sliced view), though it is not documented.
Knowing the above, we can understand why a.data_ptr() is different from b.data_ptr(). Consider the following code:
import torch

a = torch.arange(4, dtype=torch.int64)
b = a[1:]
b.data_ptr() - a.data_ptr()
# 8

The address of the first element of b is 8 more than the first element of a because we sliced to remove the first element, and each element is 8 bytes (the dtype is 64-bit integer).
If we use the same code as above but use an 8-bit integer data type, the memory address will be different by one.


Instruction: pytorch multiple branches of a model

Input:
Hi I'm trying to make this model using pytorch.
Each input is consisted of 20 images of size 28 X 28, which is C1 ~ Cp in the image.
Each image goes to CNN of same structure, but their outputs are concatenated eventually.
I'm currently struggling with feeding multiple inputs to each of its respective CNN model.
Each model in the first box with three convolutional layers will look like this as a code, but I'm not quite sure how I can put 20 different input to separate models of same structure to eventually concatenate.
        self.features = nn.Sequential(
            nn.Conv2d(1,10, kernel_size = 3, padding = 1),
            nn.ReLU(),
            nn.Conv2d(10, 14, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(14, 18, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Flatten(),
            nn.Linear(28*28*18, 256)
        )

I've tried out giving a list of inputs as an input to forward function, but it ended up with an error and won't go through.
I'll be more than happy to explain further if anything is unclear.

Output:
Simply define forward as taking a list of tensors as input, then process each input with the corresponding CNN (in the example snippet, CNNs share the same structure but don't share parameters, which is what I assume you need. You'll need to fill in the dots ... according to your specifications.
class MyModel(torch.nn.Module):
   def __init__(self, ...):
       ...
       self.cnns = torch.nn.ModuleList([torch.nn.Sequential(...) for _ in range(20)])

   def forward(xs: list[Tensor]):
       return torch.cat([cnn(x) for x, cnn in zip(xs, self.cnns)], dim=...)


Instruction: How to feed a image to Generator in GAN Pytorch

Input:
So, I'm training a DCGAN model in pytorch on celeba dataset (people). And here is the architecture of the generator:
Generator(
  (main): Sequential(
    (0): ConvTranspose2d(100, 512, kernel_size=(4, 4), stride=(1, 1), bias=False)
    (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
    (3): ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (4): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): ReLU(inplace=True)
    (6): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (7): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (8): ReLU(inplace=True)
    (9): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (11): ReLU(inplace=True)
    (12): ConvTranspose2d(64, 3, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (13): Tanh()
  )
)

So after training, I want to check what generator outputs if I feed an occluded image like this:
 (size: 64X64)
But as u might have guessed that the image has 3 channels and my generator accepts a latent vector of 100 channels at the starting, so what is the correct way to feed this image to the generator and check the output. (I'm expecting that the generator tries to generate only the occluded part of the image). If you want a reference code then try this demo file of pytorch. I have modified this file according to my own needs, so for referring, this will do the trick.

Output:
You just can't do that. As you said, your network expects 100 dimensional input which is normally sampled from standard normal distribution:

So the generator's job is to take this random vector and generate 3x64x64 image that is indistinguishable from real images. Input is a random 100 dimensional vector sampled from standard normal distribution. I don't see any way to input your image into the current network without modifying the architecture and retraining the new model. If you want to try a new model, you can change input to occluded images, apply some conv. / linear layers to reduce the dimensions to 100 then keep the rest of the network same. This way network will try to learn to generate images not from latent vector but from the feature vector extracted from occluded images. It may or may not work.
EDIT I've decided to give it a go and see if network can learn with this type of conditioned input vectors instead of latent vectors. I've used the tutorial example you've linked and added a couple of changes. First a new network for receiving input and reducing it to 100 dimensions:
class ImageTransformer(nn.Module):
    def __init__(self):
        super(ImageTransformer, self).__init__()
        self.main = nn.Sequential(
            nn.Conv2d(3, 1, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True)
        )

        self.linear = nn.Linear(32*32, 100)

    def forward(self, input):
        out = self.main(input).view(input.shape[0], -1)
        return self.linear(out).view(-1, 100, 1, 1)

Just a simple convolution layer + relu + linear layer to map to 100 dimensions at the output. Note that you can try a much better network here as a better feature extractor, I just wanted to make a simple test.
fixed_input = next(iter(dataloader))[0][0:64, :, : ,:]
fixed_input[:, :, 20:44, 20:44] = torch.tensor(np.zeros((24,24), dtype = np.float32))
fixed_input = fixed_input.to(device)

This is how I modify the tensor to add a black patch over the input. Just sampled a batch to create a fixed input to track the process as it was done in the tutorial with a random vector.
# Create the generator
netG = Generator().to(device)
netD = Discriminator().to(device)
netT = ImageTransformer().to(device)

# Apply the weights_init function to randomly initialize all weights
#  to mean=0, stdev=0.2.
netG.apply(weights_init)
netD.apply(weights_init)
netT.apply(weights_init)

# Print the model
print(netG)
print(netD)
print(netT)

Most of the steps are same, just created an instance of the new transformer network. Then finally, training loop is slightly modified where generator is not fed random vectors but it is given outputs of the new transformer network.
img_list = []
G_losses = []
D_losses = []
iters = 0

for epoch in range(num_epochs):
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        transformed = data[0].detach().clone()
        transformed[:, :, 20:44, 20:44] = torch.tensor(np.zeros((24,24), dtype = np.float32))
        transformed = transformed.to(device)
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
        output = netD(real_cpu).view(-1)
        errD_real = criterion(output, label)
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        fake = netT(transformed)
        fake = netG(fake)
        label.fill_(fake_label)
        output = netD(fake.detach()).view(-1)
        errD_fake = criterion(output, label)
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        errD = errD_real + errD_fake
        optimizerD.step()

        ############################
        # (2) Update G network: maximize log(D(G(z)))
        ###########################
        netG.zero_grad()
        label.fill_(real_label)
        output = netD(fake).view(-1)
        errG = criterion(output, label)
        errG.backward()
        D_G_z2 = output.mean().item()
        optimizerG.step()

        # Output training stats
        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

        # Save Losses for plotting later
        G_losses.append(errG.item())
        D_losses.append(errD.item())

        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netT(fixed_input)
                fake = netG(fake).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1

Training was somewhat okay in terms of loss reductions etc. Finally this is what I got after 5 epochs training:

So what does this result tell us? Since the generator's inputs were not randomly taken from a normal distribution, generator wasn't able to learn the distribution of faces to create varying range of output faces. And since the input is a conditioned feature vector, output images' range is limited. So in summary, random inputs are required for the generator even though it learned to remove patches :)


Instruction: Issue while using transformers package inside the docker image

Input:
I am using transformers pipeline to perform sentiment analysis on sample texts from 6 different languages. I tested the code in my local Jupyterhub and it worked fine. But when I wrap it in a flask application and create a docker image out of it, the execution is hanging at the pipeline inference line and its taking forever to return the sentiment scores.

mac os catalina 10.15.7 (no GPU)
Python version : 3.8
Transformers package : 4.4.2
torch version : 1.6.0

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
classifier = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)
results = classifier(["We are very happy to show you the Transformers library.", "We hope you don't hate it."])
print([i['score'] for i in results])

The above code works fine in Jupyter notebook and it has provided me the expected result
[0.7495927810668945,0.2365245819091797]

So now if I create a docker image with flask wrapper its getting stuck at the results = classifier([input_data]) line and the execution is running forever.
My folder structure is as follows:
- src
    |-- app
         |--main.py
    |-- Dockerfile
    |-- requirements.txt

I used the below Dockerfile to create the image
FROM tiangolo/uwsgi-nginx-flask:python3.8
COPY ./requirements.txt /requirements.txt
COPY ./app /app
WORKDIR /app
RUN pip install -r /requirements.txt
RUN echo "uwsgi_read_timeout 1200s;" > /etc/nginx/conf.d/custom_timeout.conf

And my requirements.txt file is as follows:
pandas==1.1.5
transformers==4.4.2
torch==1.6.0

My main.py script look like this :
from flask import Flask, json, request, jsonify
import traceback
import pandas as pd
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

app = Flask(__name__)
app.config["JSON_SORT_KEYS"] = False

model_name = 'nlptown/bert-base-multilingual-uncased-sentiment'
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
nlp = pipeline('sentiment-analysis', model=model_path, tokenizer=model_path)

@app.route("/")
def hello():
    return "Model: Sentiment pipeline test"

@app.route("/predict", methods=['POST'])
def predict():

    json_request = request.get_json(silent=True)
    input_list = [i['text'] for i in json_request["input_data"]]

    results = nlp(input_list)         ##########  Getting stuck here
    for result in results:
        print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
    score_list = [round(i['score'], 4) for i in results]

    return jsonify(score_list)

if __name__ == "__main__":
    app.run(host='0.0.0.0', debug=False, port=80)

My input payload is of the form
{"input_data" : [{"text" : "We are very happy to show you the Transformers library."},
                 {"text" : "We hope you don't hate it."}]}

I tried looking into the transformers github issues but couldn't find one. I execution works fine even when using the flask development server but it runs forever when I wrap it and create a docker image. I am not sure if I am missing any additional dependency to be included while creating the docker image.
Thanks.

Output:
I was having a similar issue. It seems that starting the app somehow polutes the memory of transformers models. Probably something to do with how Flask does threading but no idea why. What fixed it for me was doing the things that are causing trouble (loading the models) in a different thread.
import threading

def preload_models():
    "LOAD MODELS"
    return 0

def start_app():

    app = create_app()
    register_handlers(app)

    preloading = threading.Thread(target=preload_models)
    preloading.start()
    preloading.join()

    return app

First reply here. I would be really glad if this helps.


Instruction: PyTorch not working when using Pytorch with cuda 11.1: Dataloader

Input:
So, I'm running this deep neural network. And normally, when i train it with the cpu using the pytorch library WITHOUT cuda, it runs fine.
However, I noticed when I installed pytorch+cuda 11.1, whenever I try to enumerate over the Dataloader, the following error gets thrown out:
OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\Name\Desktop\Folder\lib\site-packages\torch\lib\cudnn_adv_train64_8.dll" or one of its dependencies.

But, when i use pip to uninstall pytorch+cuda and install pytorch without cuda, my script runs fine.
both versions of pytorch are 1.8.0
Does anyone know how to go about fixing this? Would love to be able to use this gpu.
Note: This is how i set up the dataloader:
    train_dataloader = DataLoader(
        train_dataset,
        batch_size=args.batch_size,
        num_workers=args.num_workers,
        shuffle=True,
        pin_memory=(device == "cuda"),
    )

Output:
See this issue on the PyTorch github page: https://github.com/Spandan-Madan/Pytorch_fine_tuning_Tutorial/issues/10
Basically you need to turn off the automatic paging file management in Windows.

Windows Key
Search for: advanced system settings
Advanced tab
Performance - "Settings" button
Advanced tab - "Change" button
Uncheck the "Automatically manage paging file size for all drives" checkbox
Select the "System managed size" radio button.
Reboot


Instruction: GPU memory increasing at each batch (PyTorch)

Input:
I am trying to build a convolutionnal network using ConvLSTM layer (LSTM cell but with convolutions instead of matrix multiplications), but the problem is that my GPU memory increases at each batch, even if I'm deleting variables, and getting the true value for the loss (and not the graph) for each iteration. I may be doing something wrong but that exact same script ran without issues with another model (with more parameters and also using ConvLSTM layer).
Each batch is composed of num_batch x 3 images (grayscale) and I'm trying to predict the difference |Im(t+1)-Im(t)| with the input Im(t)
def main():
    config = Config()

    train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=config.batch_size, num_workers=0, shuffle=True, drop_last=True)

    nb_img = len(train_dataset)
    util.clear_progress_dir()

    step_tensorboard = 0
    ###################################
    #          Model Setup            #
    ###################################

    model = fully_convLSTM()
    if torch.cuda.is_available():
        model = model.float().cuda()

    lr = 0.001
    optimizer = torch.optim.Adam(model.parameters(),lr=lr)

    util.enumerate_params([model])

    ###################################
    #          Training Loop          #
    ###################################

    model.train() #Put model in training mode

    train_loss_recon = []
    train_loss_recon2 = []

    for epoch in tqdm(range(config.num_epochs)):

        running_loss1 = 0.0
        running_loss2 = 0.0

        for i, (inputs, outputs) in enumerate(train_dataloader, 0):
            print(i)
            torch.cuda.empty_cache()
            gc.collect()

           # if torch.cuda.is_available():
            inputs  = autograd.Variable(inputs.float()).cuda()
            outputs = autograd.Variable(outputs.float()).cuda()

            im1 =  inputs[:,0,:,:,:]
            im2 =  inputs[:,1,:,:,:]
            im3 =  inputs[:,2,:,:,:]

            diff1 = torch.abs(im2 - im1).cuda().float()
            diff2 = torch.abs(im3 - im2).cuda().float()

            model.initialize_hidden()

            optimizer.zero_grad()
            pred1 = model.forward(im1)
            loss = reconstruction_loss(diff1, pred1)
            loss.backward()
            # optimizer.step()

            model.update_hidden()

            optimizer.zero_grad()
            pred2 = model.forward(im2)
            loss2 = reconstruction_loss(diff2, pred2)
            loss2.backward()
            optimizer.step()

            model.update_hidden()

            ## print statistics

            running_loss1 += loss.detach().data
            running_loss2 += loss2.detach().data

            if i==0:

                with torch.no_grad():
                    img_grid_diff_true = (diff2).cpu()
                    img_grid_diff_pred = (pred2).cpu()

                    f, axes = plt.subplots(2, 4, figsize=(48,48))
                    for l in range(4):
                        axes[0, l].imshow(img_grid_diff_true[l].squeeze(0).squeeze(0), cmap='gray')
                        axes[1, l].imshow(img_grid_diff_pred[l].squeeze(0).squeeze(0), cmap='gray')

                    plt.show()
                    plt.close()

                    writer_recon_loss.add_scalar('Reconstruction loss', running_loss1, step_tensorboard)
                    writer_recon_loss2.add_scalar('Reconstruction loss2', running_loss2, step_tensorboard)

                    step_tensorboard += 1

            del pred1
            del pred2
            del im1
            del im2
            del im3
            del diff1
            del diff2#, im1_noised, im2_noised
            del inputs
            del outputs
            del loss
            del loss2
            for obj in gc.get_objects():
                if torch.is_tensor(obj) :
                    del obj

            torch.cuda.empty_cache()
            gc.collect()

        epoch_loss = running_loss1 / len(train_dataloader.dataset)
        epoch_loss2 = running_loss2/ len(train_dataloader.dataset)
        print(f"Epoch {epoch} loss reconstruction1: {epoch_loss:.6f}")
        print(f"Epoch {epoch} loss reconstruction2: {epoch_loss2:.6f}")

        train_loss_recon.append(epoch_loss)
        train_loss_recon2.append(epoch_loss2)

        del running_loss1, running_loss2, epoch_loss, epoch_loss2

Here is the model used :
class ConvLSTMCell(nn.Module):
    def __init__(self, input_channels, hidden_channels, kernel_size):
        super(ConvLSTMCell, self).__init__()

        # assert hidden_channels % 2 == 0

        self.input_channels = input_channels
        self.hidden_channels = hidden_channels
        self.kernel_size = kernel_size
        # self.num_features = 4

        self.padding = 1

        self.Wxi = nn.Conv2d(self.input_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=True)
        self.Whi = nn.Conv2d(self.hidden_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=False)
        self.Wxf = nn.Conv2d(self.input_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=True)
        self.Whf = nn.Conv2d(self.hidden_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=False)
        self.Wxc = nn.Conv2d(self.input_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=True)
        self.Whc = nn.Conv2d(self.hidden_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=False)
        self.Wxo = nn.Conv2d(self.input_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=True)
        self.Who = nn.Conv2d(self.hidden_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=False)

        self.Wci = None
        self.Wcf = None
        self.Wco = None

    def forward(self, x, h, c): ## Equation (3) dans Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
        ci = torch.sigmoid(self.Wxi(x) + self.Whi(h) + c * self.Wci)
        cf = torch.sigmoid(self.Wxf(x) + self.Whf(h) + c * self.Wcf)
        cc = cf * c + ci * torch.tanh(self.Wxc(x) + self.Whc(h)) ###gt= tanh(cc)
        co = torch.sigmoid(self.Wxo(x) + self.Who(h) + cc * self.Wco) ##channel out = hidden channel
        ch = co * torch.tanh(cc)
        return ch, cc #short memory, long memory

    def init_hidden(self, batch_size, hidden, shape):
        if self.Wci is None:
            self.Wci = nn.Parameter(torch.zeros(1, hidden, shape[0], shape[1])).cuda()
            self.Wcf = nn.Parameter(torch.zeros(1, hidden, shape[0], shape[1])).cuda()
            self.Wco = nn.Parameter(torch.zeros(1, hidden, shape[0], shape[1])).cuda()
        else:
            assert shape[0] == self.Wci.size()[2], 'Input Height Mismatched!'
            assert shape[1] == self.Wci.size()[3], 'Input Width Mismatched!'
        return (autograd.Variable(torch.zeros(batch_size, hidden, shape[0], shape[1])).cuda(),
                autograd.Variable(torch.zeros(batch_size, hidden, shape[0], shape[1])).cuda())

class fully_convLSTM(nn.Module):
    def __init__(self):
        super(fully_convLSTM, self).__init__()
        layers = []
        self.hidden_list = [1,32,32,1]#,32,64,32,
        for k in range(len(self.hidden_list)-1):   # Define blocks of [ConvLSTM,BatchNorm,Relu]
            name_conv = "self.convLSTM" +str(k)
            cell_conv = ConvLSTMCell(self.hidden_list[k],self.hidden_list[k+1],3)
            setattr(self, name_conv, cell_conv)
            name_batchnorm = "self.batchnorm"+str(k)
            batchnorm=nn.BatchNorm2d(self.hidden_list[k+1])
            setattr(self, name_batchnorm, batchnorm)
            name_relu =" self.relu"+str(k)
            relu=nn.ReLU()
            setattr(self, name_relu, relu)
        self.sigmoid = nn.Sigmoid()

        self.internal_state=[]

    def initialize_hidden(self):
        for k in range(len(self.hidden_list)-1):
            name_conv = "self.convLSTM" +str(k)
            (h,c) = getattr(self,name_conv).init_hidden(config.batch_size, self.hidden_list[k+1],(256,256))
            self.internal_state.append((h,c))
        self.internal_state_new=[]
    def update_hidden(self):
        for i, hidden in enumerate(self.internal_state_new):
            self.internal_state[i] = (hidden[0].detach(), hidden[1].detach())
        self.internal_state_new = []
    def forward(self, input):
        x = input
        for k in range(len(self.hidden_list)-1):
            name_conv = "self.convLSTM" +str(k)
            name_batchnorm = "self.batchnorm"+str(k)
            name_relu =" self.relu"+str(k)
            x, c = getattr(self,name_conv)(x, self.internal_state[k][1], self.internal_state[k][0])
            self.internal_state_new.append((x.detach(),c.detach()))
            x = getattr(self,name_batchnorm)(x)
            if k!= len(self.hidden_list)-2:
                x = getattr(self,name_relu)(x)
            else :
                x = self.sigmoid(x)
        return x

So my question is, what in my code is causing memory to accumulate during the training phase?

Output:
A few quick notes about training code:

torch.Variable is deprecated since at least 8 minor versions (see here), don't use it
gc.collect() has no point, PyTorch does the garbage collector on it's own
Don't use torch.cuda.empty_cache() for each batch, as PyTorch reserves some GPU memory (doesn't give it back to OS) so it doesn't have to allocate it for each batch once again. It will make your code slow, don't use this function at all tbh, PyTorch handles this.
Don't spam random memory cleaning, that's most probably not where the error is

Model
Yes, this is probably the case (although it's hard to read this model's code).
Take notice of self.internal_state list and self.internal_state_new list also.

Each time you call model.initialize_hidden() a new set of tensor is added to this list (and never cleaned as far as I can tell)
self.internal_state_new seems to be cleaned in update_hidden, maybe self.internal_state should be also?

In essence, check out this self.internal_state property of your model, the list grows indefinitely from what I see. Initializing with zeros everywhere is quite strange, there is probably no need to do that (e.g. PyTorch's RNN is initialized with zeros by default, this is probably similar).


Instruction: Confusion about the shape of the output logits from Resnet

Input: I am trying to understand why the shape of the output logits from the Resnet18 model I am working with are (27, 19). The shape of 19 I understand, that is the number of classes I have set the model to predict, but the shape of 27 is the part that I am confused about. I have a batch size of 64 so I would have thought the shape of the logits would be (64, 19), because that would give me 1 prediction vector for each image in the batch...

Output: Turns out I was looking at the logits from the last batch in my epoch, and there weren't enough images left to fill up the entire 64 batch size so it only has 27 images left to train on.


Instruction: Pytorch1.6 What is the actual learning rate during training?

Input:
I'd like to know the actual learning rate during training, here is my code.
learning_rate = 0.001
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[1, 2], gamma=0.1)

def train(epoch):
    train_loss = 0

    for batch_idx, (input, target) in enumerate(train_loader):
        predict_label = net(input)
        loss = criterion(predict_label, target)
        train_loss += loss.item()

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print(optimizer.param_groups[0]['lr'])

    scheduler.step()
    print(scheduler.state_dict()['_last_lr'])
    print(optimizer.param_groups[0]['lr'])

the output is 0.001, 0.0001, 0.0001. So what is the actual lr during optimizer.step()? 0.001 or 0.0001? Thanks.

Output:
The important part is here:
for batch_idx, (input, target) in enumerate(train_loader):
    ...

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print(optimizer.param_groups[0]['lr']) #### CURRENT LEARNING RATE

scheduler.step() #step through learning rate
print(scheduler.state_dict()['_last_lr']) #### NEW LEARNING RATE
print(optimizer.param_groups[0]['lr']) #### NEW LEARNING RATE

Because you step your scheduler after your epoch, then the first epoch will have your initial value which is set to 0.001. If you run for multiple epochs then it will continue to be annealed.


Instruction: Trouble implementing "concurrent" softmax function from paper (PyTorch)

Input:
I am trying to implement the so called 'concurrent' softmax function given in the paper "Large-Scale Object Detection in the Wild from Imbalanced Multi-Labels". Below is the definition of the concurrent softmax:

NOTE: I have left the (1-rij) term out for the time being because I don't think it applies to my problem given that my training dataset has a different type of labeling compared to the paper.

To keep it simple for myself I am starting off by implementing it in a very inefficient, but easy to follow, way using for loops. However, the output I get seems wrong to me. Below is the code I am using:
# here is a one-hot encoded vector for the multi-label classification
# the image thus has 2 correct labels out of a possible 3 classes
y = [0, 1, 1]

# these are some made up logits that might come from the network.
vec = torch.tensor([0.2, 0.9, 0.7])

def concurrent_softmax(vec, y):
    for i in range(len(vec)):
        zi = torch.exp(vec[i])
        sum_over_j = 0
        for j in range(len(y)):
            sum_over_j += (1-y[j])*torch.exp(vec[j])

        out = zi / (sum_over_j + zi)
        yield out

for result in concurrent_softmax(vec, y):
    print(result)

From this implementation I have realized that, no matter what value I give to the first logit in 'vec' I will always get an output of 0.5 (because it essentially always calculates zi / (zi+zi)). This seems like a major problem, because I would expect the value of the logits to have some influence on the resulting concurrent-softmax value. Is there a problem in my implementation then, or is this behaviour of the function correct and there is something theoretically that I am not understanding?

Output:
This is the expected behaviour given y[i]=1 for all other i.
Note you can simplify the summation with a dot product:
y = torch.tensor(y)

def concurrent_softmax(z, y):
    sum_over_j = torch.dot((torch.ones(len(y)) - y), torch.exp(z))

    for zi in z:
        numerator = torch.exp(zi)
        denominator = sum_over_j + numerator
        yield numerator / denominator


Instruction: Build a pytorch model wrap around another pytorch model

Input:
Is it possible to wrap a pytorch model inside another pytorch module? I could not do it the normal way like in transfer learning (simply concatenating some more layers) because in order to get the intended value for the next 'layer', I need to wait the last layer of the first module to generate multiple outputs (say 100) and to use all those outputs to get the value for the next 'layer' (say taking the max of those outputs). I tried to define the integrated model as something like the following:
class integrated(nn.Module):
    def __init__(self):
        super(integrated, self)._init_()

    def forward(self, x):

        model = VAE(
        encoder_layer_sizes=args.encoder_layer_sizes,
        latent_size=args.latent_size,
        decoder_layer_sizes=args.decoder_layer_sizes,
        conditional=args.conditional,
        num_labels=10 if args.conditional else 0).to(device)
        device = torch.device('cpu')
        model.load_state_dict(torch.load(r'...'))  # the first model is saved somewhere else beforehand
        model.eval()

        temp = []
        for j in range(100):
                x = model(x)
                temp.append(x)

        y=max(temp)

        return y

The reason I would like to do that is the library I need to use requires the input itself to be a pytorch module. Otherwise I could simply leave the last part outside of the module.

Output:
Yes you can definitely use a Pytorch module inside another Pytorch module. The way you are doing this in your example code is a bit unusual though, as external modules (VAE, in your case) are more often initialized in the __init__ function and then saved as attributes of the main module (integrated). Among other things, this avoids having to reload the sub-module every time you call forward.
One other thing that looks a bit funny is your for loop over repeated invocations of model(x). If there is no randomness involved in model's evaluation, then you would only need a single call to model(x), since all 100 calls will give the same value. So assuming there is some randomness, you should consider whether you can get the desired effect by batching together 100 copies of x and using a single call to model with this batched input. This ultimately depends on additional information about why you are calling this function multiple times on the same input, but either way, using a single batched evaluation will be a lot faster than using many unbatched evaluations.


Instruction: PyTorch ToTensor scaling to [0,1] discrepancy

Input:
If I do this
mnist_train = MNIST('../data/MNIST', download = True,
                transform = transforms.Compose([
                    transforms.ToTensor(),
                ]), train = True)

and
mnist_train.data.max()
why do I get 255? I should get 1, because ToTensor() scales to [0,1], right?
If  I do:
for i in range(0, len(mnist_train)):
    print(mnist_train[i][0].max())

then, I get almost 1?
Could someone please help me understand this?

Output:
When you do
mnist_train.data

PyTorch gives you the data attribute of the mnist_train, which is defined on this line (when you make a MNIST instance). And if you look at the codes before it in the __init__, no transformation happens!
OTOH, when you do
mnist_train[i]

the __getitem__ method of the object is triggered which you can find here.  There is an if statement for transform in this method and therefore you get the transformed version now.
Since a common usage is using this MNIST dataset (or any other one) through torch.utils.data.DataLoader and it calls this __getitem__, we get the normalized values.


Instruction: BERT: Weights of input embeddings as part of the Masked Language Model

Input:
I looked through different implementations of BERT's Masked Language Model.
For pre-training there are two common versions:

Decoder would simply take the final embedding of the [MASK]ed token and pass it throught a linear layer (without any modifications):

    class LMPrediction(nn.Module):
        def __init__(self, hidden_size, vocab_size):
            super().__init__()
            self.decoder = nn.Linear(hidden_size, vocab_size, bias = False)
            self.bias = nn.Parameter(torch.zeros(vocab_size))
            self.decoder.bias = self.bias
        def forward(self, x):
             return self.decoder(x)

Some implementations would use the weights of the input embeddings as weights of the decoder-linear-layer:

    class LMPrediction(nn.Module):
        def __init__(self, hidden_size, vocab_size, embeddings):
            super().__init__()
            self.decoder = nn.Linear(hidden_size, vocab_size, bias = False)
            self.bias = nn.Parameter(torch.zeros(vocab_size))
            self.decoder.weight = embeddings.weight ## <- THIS LINE
            self.decoder.bias = self.bias
        def forward(self, x):
             return self.decoder(x)

Which one is correct? Mostly, I see the first implementation. However, the second one makes sense as well - but I cannot find it mentioned in any papers (I would like to see if the second version is somehow superior to the first one)

Output:
For those who are interested, it is called weight tying or joint input-output embedding. There are two papers that argue for the benefit of this approach:

Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation
Using the Output Embedding to Improve Language Models


Instruction: Pytorch gather question (3D Computer Vision)

Input:
I have N groups of C-dimension points. In each groups there are M points. So, there is a tensor of (N, M, C). Let's call it features.
I calculated the maximum element and the index through M dimension, to find the maximum points for each C dimension (a max pooling operation), resulting max tensor (N, 1, C) and index tensor (N, 1, C).
I have another tensor of shape (N, M, 3) storing the geometric coordinates of those N*M high-dimention points. Now, I want to use the index from the maximum points in each C dimension, to get the coordinates of all those maximum points.
For example, N=2, M=4, C=6.
The coordinate tensor, whose shape is (2, 4, 3):
[[[1, 2, 3]
  [4, 5, 6]
  [7, 8, 9]
  [8, 7, 6]]

 [11, 12, 13]
 [14, 15, 16]
 [17, 18, 19]
 [18, 17, 16]]]

The indices tensor, whose shape is (2, 1, 6):
[[[0, 1, 2, 1, 2, 3]]
 [[1, 2, 3, 2, 1, 0]]]

For example, the first element in indices is 0, I want to grab [1, 2, 3] from the coordinate tensor out. For the second element (1), I want to grab [4, 5, 6] out. For the third element in the next dimension (3), I want to grab [18, 17, 16] out.
The result tensor will look like:
[[[1, 2, 3]  # 0
  [4, 5, 6]  # 1
  [7, 8, 9]  # 2
  [4, 5, 6]  # 1
  [7, 8, 9]  # 2
  [8, 7, 6]] # 3

 [[14, 15, 16] # 1
  [17, 18, 19] # 2
  [18, 17, 16] # 3
  [17, 18, 19] # 2
  [14, 15, 16] # 1
  [11, 12, 13]]]# 0

whose shape is (2, 6, 3).
I tried to use torch.gather but I can not get it worked. I wrote a naive algorithm enumerating all N groups, but indeed it is slow, even using TorchScript's JIT. So, how to write this efficiently in pytorch?

Output:
You can use integer array indexing combined with broadcasting semantics to get your result.
import torch

x = torch.tensor([
    [[1, 2, 3],
     [4, 5, 6],
     [7, 8, 9],
     [8, 7, 6]],
    [[11, 12, 13],
     [14, 15, 16],
     [17, 18, 19],
     [18, 17, 16]],
])

i = torch.tensor([[[0, 1, 2, 1, 2, 3]],
                  [[1, 2, 3, 2, 1, 0]]])

# rows is shape [2, 1], cols is shape [2, 6]
rows = torch.arange(x.shape[0]).type_as(i).unsqueeze(1)
cols = i.squeeze(1)

# y is [2, 6, ...]
y = x[rows, cols]


Instruction: Huggingface error: AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'

Input:
I am trying to tokenize some numerical strings using a WordLevel/BPE tokenizer, create a data collator and eventually use it in a PyTorch DataLoader to train a new model from scratch.
However, I am getting an error

AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'

when running the following code
from transformers import DataCollatorForLanguageModeling
from tokenizers import ByteLevelBPETokenizer
from tokenizers.pre_tokenizers import Whitespace
from torch.utils.data import DataLoader, TensorDataset

data = ['4814 4832 4761 4523 4999 4860 4699 5024 4788 <unk>']

# Tokenizer
tokenizer = ByteLevelBPETokenizer()
tokenizer.pre_tokenizer = Whitespace()
tokenizer.train_from_iterator(data, vocab_size=1000, min_frequency=1,
    special_tokens=[
        "<s>",
        "</s>",
        "<unk>",
        "<mask>",
    ])

# Data Collator
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False
)

train_dataset = TensorDataset(torch.tensor(tokenizer(data, ......)))

# DataLoader
train_dataloader = DataLoader(
    train_dataset,
    collate_fn=data_collator
)

Is this error due to not having configured the pad_token_id for the tokenizer? If so, how can we do this?
Thanks!
Error trace:
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/anaconda3/envs/x/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/anaconda3/envs/x/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/opt/anaconda3/envs/x/lib/python3.8/site-packages/transformers/data/data_collator.py", line 351, in __call__
    if self.tokenizer.pad_token_id is not None:
AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'

Conda packages
pytorch                   1.7.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
pytorch-lightning         1.2.5              pyhd8ed1ab_0    conda-forge
tokenizers                0.10.1                   pypi_0    pypi
transformers              4.4.2                    pypi_0    pypi

Output:
The error tells you that the tokenizer needs an attribute called pad_token_id. You can either wrap the ByteLevelBPETokenizer into a class with such an attribute (... and met other missing attributes down the road) or use the wrapper class from the transformers library:
from transformers import PreTrainedTokenizerFast

#your code
tokenizer.save(SOMEWHERE)
tokenizer = PreTrainedTokenizerFast(tokenizer_file=tokenizer_path)


Instruction: where should I change the code to tell PyTorch not to use the GPU?

Input:
As mentioned in How to tell PyTorch to not use the GPU?, in order to tell PyTorch not to use the GPU you should change a few lines inside PyTorch code.
Where should I make the change?
Where is the line of code that needs to be modified?
I tried to find it but couldn't...

Output:
Using the method .cpu() on any tensor or pytorch module transfers the component to the cpu so that the calculations will be made using it.
Another direction is to use the method .to("cpu"). Alternatively, you can change "cpu" with the name of other devices such as "cuda".
Example:
a)
model = MyModel().cpu()  # move the model to the cpu
x = data.cpu()  # move the input to the cpu
y = model(x)

b)
model = MyModel().to('cpu')  # move the model to the cpu
x = data.to('cpu')  # move the input to the cpu
y = model(x)


Instruction: Pytorch Model Summary

Input:
I am trying to load a pytorch model using:
model = torch.load('/content/gdrive/model.pth.tar', map_location='cpu')

I want to check the model summary. When I try:
print(model)

I get the following output:
{'state_dict': {'model.conv1.weight': tensor([[[[ 2.0076e-02,  1.5264e-02, -1.2309e-02,  ..., -4.0222e-02,
           -4.0527e-02, -6.4458e-02],
          [ 6.3291e-03,  3.8393e-03,  1.2400e-02,  ..., -3.3926e-03,
           -2.1063e-02, -3.4743e-02],
          [ 1.9969e-02,  2.0064e-02,  1.4004e-02,  ...,  8.7359e-02,
            5.4801e-02,  4.8791e-02],
          ...,
          [ 2.5362e-02,  1.1433e-02, -7.6776e-02,  ..., -3.4798e-01,
           -2.7198e-01, -1.2066e-01],
          [ 8.0373e-02,  1.3095e-01,  1.4240e-01,  ..., -2.2933e-03,
           -1.0469e-01, -1.0922e-01],
          [-1.1147e-03,  7.4572e-02,  1.2814e-01,  ...,  1.6903e-01,
            1.0619e-01,  2.4744e-02]],
      'model.layer4.1.bn2.running_var': tensor([0.0271, 0.0155, 0.0199, 0.0198, 0.0132, 0.0148, 0.0182, 0.0170, 0.0134,
.
.
.

What does it even mean?
I also tried to use:
from torchsummary import summary
summary(model, input_size=(3, 224, 224))

But it gives me the following error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-31-ca828d30bd38> in <module>()
      1 from torchsummary import summary
----> 2 summary(model, input_size=(3, 224, 224))

/usr/local/lib/python3.7/dist-packages/torchsummary/torchsummary.py in summary(model, input_size, batch_size, device)
     66
     67     # register hook
---> 68     model.apply(register_hook)
     69
     70     # make a forward pass

AttributeError: 'dict' object has no attribute 'apply'

Please note that model is a custom model that I am trying to load.
How can I get a model summary in Pytorch?

Output:
You loaded the "*.pt" and didn't feed it to a model (which is just a dictionary of the weights depending on what you saved) this is why you get the following output:
{'state_dict': {'model.conv1.weight': tensor([[[[ 2.0076e-02,  1.5264e-02, -1.2309e-02,  ..., -4.0222e-02,
           -4.0527e-02, -6.4458e-02],
          [ 6.3291e-03,  3.8393e-03,  1.2400e-02,  ..., -3.3926e-03,
           -2.1063e-02, -3.4743e-02],
          [ 1.9969e-02,  2.0064e-02,  1.4004e-02,  ...,  8.7359e-02,
            5.4801e-02,  4.8791e-02],
          ...,
          [ 2.5362e-02,  1.1433e-02, -7.6776e-02,  ..., -3.4798e-01,
           -2.7198e-01, -1.2066e-01],
          [ 8.0373e-02,  1.3095e-01,  1.4240e-01,  ..., -2.2933e-03,
           -1.0469e-01, -1.0922e-01],
          [-1.1147e-03,  7.4572e-02,  1.2814e-01,  ...,  1.6903e-01,
            1.0619e-01,  2.4744e-02]],
      'model.layer4.1.bn2.running_var': tensor([0.0271, 0.0155, 0.0199, 0.0198, 0.0132, 0.0148, 0.0182, 0.0170, 0.0134,
.
.
.

What you should do is:
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
print(model)

You can refer to the pytorch doc
Regarding your second attempt, the same issue causing the problem, summary expect a model and not a dictionary of the weights.


Instruction: Pytorch embedding too big for GPU but fits in CPU

Input:
I am using PyTorch lightning, so lightning control GPU/CPU assignments and in
return I get easy multi GPU support for training.
I would like to create an embedding that does not fit in the GPU memory.
fit_in_cpu = torch.nn.Embedding(too_big_for_GPU, embedding_dim)

Then when I select the subset for a batch, send it to the GPU
GPU_tensor = embedding(idx)

How do I do this in Pytorch Lightning?

Output:
Lightning will send anything that is registered as a model parameter to GPU, i.e: weights of layers (anything in torch.nn.*) and variables registered using torch.nn.parameter.Parameter.
However if you want to declare something in CPU and then on runtime move it to GPU you can go 2 ways:

Create the too_big_for_GPU inside the __init__ without registering it as a model parameter (using torch.zeros or torch.randn or any other init function). Then move it to the GPU on the forward pass

class MyModule(pl.LightningModule):
    def __init__():
        self.too_big_for_GPU = torch.zeros(4, 1000, 1000, 1000)
    def forward(self, x):
        # Move tensor to same GPU as x and operate with it
        y = self.too_big_for_GPU.to(x.device) * x**2
        return y

Create the too_big_for_GPU which will be created by default in CPU and then you would need to move it to GPU

class MyModule(pl.LightningModule):
    def forward(self, x):
        # Create the tensor on the fly and move it to x GPU
        too_big_for_GPU = torch.zeros(4, 1000, 1000, 1000).to(x.device)
        # Operate with it
        y = too_big_for_GPU * x**2
        return y


Instruction: How to quickly inverse a permutation by using PyTorch?

Input:
I am confused on how to quickly restore an array shuffled by a permutation.
Example #1:

[x, y, z] shuffled by P: [2, 0, 1], we will obtain [z, x, y]
the corresponding inverse should be P^-1: [1, 2, 0]

Example #2:

[a, b, c, d, e, f] shuffled by P: [5, 2, 0, 1, 4, 3], then we will get [f, c, a, b, e, d]
the corresponding inverse should be P^-1: [2, 3, 1, 5, 4, 0]

I wrote the following codes based on matrix multiplication (the transpose of permutation matrix is its inverse), but this approach is too slow when I utilize it on my model training. Does there exisits a faster implementation?
import torch

n = 10
x = torch.Tensor(list(range(n)))
print('Original array', x)

random_perm_indices = torch.randperm(n).long()
perm_matrix = torch.eye(n)[random_perm_indices].t()
x = x[random_perm_indices]
print('Shuffled', x)

restore_indices = torch.Tensor(list(range(n))).view(n, 1)
restore_indices = perm_matrix.mm(restore_indices).view(n).long()
x = x[restore_indices]
print('Restored', x)

Output:
I obtained the solution in PyTorch Forum.
>>> import torch
>>> torch.__version__
'1.7.1'
>>> p1 = torch.tensor ([2, 0, 1])
>>> torch.argsort (p1)
tensor([1, 2, 0])
>>> p2 = torch.tensor ([5, 2, 0, 1, 4, 3])
>>> torch.argsort (p2)
tensor([2, 3, 1, 5, 4, 0])

Update:
The following solution is more efficient due to its linear time complexity:
def inverse_permutation(perm):
    inv = torch.empty_like(perm)
    inv[perm] = torch.arange(perm.size(0), device=perm.device)
    return inv


Instruction: How can torch.cat only have one tensor?

Input:
I think there is an error in line 53 of the following code:
CycleGan, Buffer for Images
It says:
return_images = torch.cat(return_images, 0)  # collect all the images and return

What do you think, what should it read correctly? After having gone throught the code, I am unfortunately not sure what this specific line is supposed to do, but I think I understand the rest until line 52.

Output:
torch.cat's first argument is expected to be a sequence of tensors rather than a single tensor. So you pass in like:
torch.cat([tensor_1, tensor_2, tensor_3])       # the right way

instead of
torch.cat(tensor_1, tensor_2, tensor_3)         # not the right way

In the code you linked, they are forming a list called return_images which contains many tensors in it.
np.concatenate has the same behaviour too and PyTorch designers probably mimicked the choice from there. ("cat" is short for "concatenate" which is hard to write!).


Instruction: Initialize weight in pytorch neural net

Input:
I have created this  neural net:
class _netD(nn.Module):
    def __init__(self, num_classes=1, nc=1, ndf=64):
        super(_netD, self).__init__()
        self.num_classes = num_classes
        # nc is number of channels
        # num_classes is number of classes
        # ndf is the number of output channel at the first layer
        self.main = nn.Sequential(
            # input is (nc) x 28 x 28
            # conv2D(in_channels, out_channels, kernelsize, stride, padding)
            nn.Conv2d(nc, ndf , 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf) x 14 x 14
            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*2) x 7 x 7
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*4) x 3 x 3
            nn.Conv2d(ndf * 4, ndf * 8, 3, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*8) x 2 x 2
            nn.Conv2d(ndf * 8, num_classes, 2, 1, 0, bias=False),
            # out size = batch x num_classes x 1 x 1
        )

        if self.num_classes == 1:
            self.main.add_module('prob', nn.Sigmoid())
            # output = probability
        else:
            pass
            # output = scores

    def forward(self, input):

        output = self.main(input)
        return output.view(input.size(0), self.num_classes).squeeze(1)

I want to loop through the different layers and apply a weight initialization depending on the type of layer. I am trying to do the following:
D = _netD()

for name, param in D.named_parameters():
    if type(param) == nn.Conv2d:
      param.weight.normal_(...)

But that is not working. Can you please help me?
Thanks

Output:
type(param) will only return the actual datatype called a parameter for any type of weight or data in the model. Because named_parameters() doesn't return anything useful in the name either when used on an nn.sequential-based model, you need to look at the modules to see which layers are specifically related to the nn.Conv2d class using isinstance as such:
for layer in D.modules():
    if isinstance(layer, nn.Conv2d):
         layer.weight.data.normal_(...)

Or, the way that is recommended by Soumith Chintala himself, actually just loop through your main module itself:
for L,layer in D.main:
    if isisntance(layer,nn.Conv2d):
         layer.weight.data.normal_(..)

I actually prefer the first because you don't have to specify the exact nn.sequential module itself, and will search all possible modules in the model, but either one should do the job for you.


Instruction: How to quickly generate a random permutation moving each element with a distance less than K?

Input:
I know the following PyTorch API can perform a global random shuffle for 1D array [0, ... , n-1]:
torch.randperm(n)

but I'm confused on how to quickly generate a random permutation, such that each element of the shuffled array satisfying:
K = 10  # should be positive
shuffled_array = rand_perm_func(n, K)  # a mysterious function
for i in range(n):
    print(abs(shuffled_array[i] - i) < K)  # should be True for each i

which means that each element is moved with a distance less than K. Does there exists fast implementations for 1D arrays and 2D arrays?

Thanks to @PM 2Ring, I wrote the following code:
import torch

# randomly produces a 1-D permutation index array,
# such that each element of the shuffled array has
# a distance less than K from its original location
def rand_perm(n, K):
    o = torch.arange(0, n)
    if K <= 1:
        return o
    while True:
        p = torch.randperm(n)
        d = abs(p - o) < K
        if bool(d.all()):
            return p

if __name__ == '__main__':
    for i in range(10):
        print(rand_perm(10, 2))

but it seems that when n is large, and K is small, the generation will take a very long time. Does there exists a more efficient implementation?

Output:
Here's a recursive generator in plain Python (i.e. not using PyTorch or Numpy) that produces permutations of range(n) satisfying the given constraint.
First, we create a list out to contain the output sequence, setting each slot in out to -1 to indicate that slot is unused. Then, for each value i we create a list avail of indices in the permitted range that aren't already occupied. For each j in avail, we set out[j] = i and recurse to place the next i. When i == n, all the i have been placed, so we've reached the end of the recursion, and out contains a valid solution, which gets propagated back up the recursion tree.
from random import shuffle

def rand_perm_gen(n, k):
    out = [-1] * n
    def f(i):
        if i == n:
            yield out
            return

        lo, hi = max(0, i-k+1), min(n, i+k)
        avail = [j for j in range(lo, hi) if out[j] == -1]
        if not avail:
            return
        shuffle(avail)
        for j in avail:
            out[j] = i
            yield from f(i+1)
            out[j] = -1

    yield from f(0)

def test(n=10, k=3, numtests=10):
    for j, a in enumerate(rand_perm_gen(n, k), 1):
        print("\n", j, a)
        for i, u in enumerate(a):
            print(f"{i}: {u} -> {(u - i)}")
        if j == numtests:
            break

test()

Typical output

 1 [1, 0, 3, 2, 6, 5, 4, 8, 9, 7]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 6 -> 2
5: 5 -> 0
6: 4 -> -2
7: 8 -> 1
8: 9 -> 1
9: 7 -> -2

 2 [1, 0, 3, 2, 6, 5, 4, 9, 8, 7]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 6 -> 2
5: 5 -> 0
6: 4 -> -2
7: 9 -> 2
8: 8 -> 0
9: 7 -> -2

 3 [1, 0, 3, 2, 6, 5, 4, 9, 7, 8]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 6 -> 2
5: 5 -> 0
6: 4 -> -2
7: 9 -> 2
8: 7 -> -1
9: 8 -> -1

 4 [1, 0, 3, 2, 6, 5, 4, 8, 7, 9]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 6 -> 2
5: 5 -> 0
6: 4 -> -2
7: 8 -> 1
8: 7 -> -1
9: 9 -> 0

 5 [1, 0, 3, 2, 6, 5, 4, 7, 8, 9]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 6 -> 2
5: 5 -> 0
6: 4 -> -2
7: 7 -> 0
8: 8 -> 0
9: 9 -> 0

 6 [1, 0, 3, 2, 6, 5, 4, 7, 9, 8]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 6 -> 2
5: 5 -> 0
6: 4 -> -2
7: 7 -> 0
8: 9 -> 1
9: 8 -> -1

 7 [1, 0, 3, 2, 6, 7, 4, 5, 9, 8]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 6 -> 2
5: 7 -> 2
6: 4 -> -2
7: 5 -> -2
8: 9 -> 1
9: 8 -> -1

 8 [1, 0, 3, 2, 6, 7, 4, 5, 8, 9]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 6 -> 2
5: 7 -> 2
6: 4 -> -2
7: 5 -> -2
8: 8 -> 0
9: 9 -> 0

 9 [1, 0, 3, 2, 5, 6, 4, 7, 9, 8]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 5 -> 1
5: 6 -> 1
6: 4 -> -2
7: 7 -> 0
8: 9 -> 1
9: 8 -> -1

 10 [1, 0, 3, 2, 5, 6, 4, 7, 8, 9]
0: 1 -> 1
1: 0 -> -1
2: 3 -> 1
3: 2 -> -1
4: 5 -> 1
5: 6 -> 1
6: 4 -> -2
7: 7 -> 0
8: 8 -> 0
9: 9 -> 0

Here's a live version running on SageMathCell.
This approach is faster than generating all permutations and filtering them, but it is still slow for large n. You can improve the speed by removing the shuffle call, in which case the yielded permutations are in lexicographic order.
If you just want a single solution, use next, eg
perm = next(rand_perm_gen(10, 3))

Note that all solutions share the same out list. So if you need to save those solutions in a list you have to copy them, eg
perms = [seq.copy() for seq in rand_perm_gen(5, 2)]


Instruction: How to pad zeros on Batch, PyTorch

Input:
Is there a better way to do this? How to pad tensor with zeros, without creating new tensor object? I need inputs to be of the same batchsize all the time, so I want to pad inputs that are smaller than batchsize with zeros. Like padding zeros in NLP when sequence length is shorter, but this is padding for batch.
Currently, I create a new tensor, but because of that, my GPU will go out of memory. I don't want to reduce batchsize by half to handle this operation.
import torch
from torch import nn

class MyModel(nn.Module):
    def __init__(self, batchsize=16):
        super().__init__()
        self.batchsize = batchsize

    def forward(self, x):
        b, d = x.shape

        print(x.shape) # torch.Size([7, 32])

        if b != self.batchsize: # 2. I need batches to be of size 16, if batch isn't 16, I want to pad the rest to zero
            new_x = torch.zeros(self.batchsize,d) # 3. so I create a new tensor, but this is bad as it increase the GPU memory required greatly
            new_x[0:b,:] = x
            x = new_x
            b = self.batchsize

        print(x.shape) # torch.Size([16, 32])

        return x

model = MyModel()
x = torch.randn((7, 32)) # 1. shape's batch is 7, because this is last batch, and I dont want to "drop_last"
y = model(x)
print(y.shape)

Output:
You can pad extra elements like so:
import torch.nn.functional as F

n = self.batchsize - b

new_x = F.pad(x, (0,0,n,0)) # pad the start of 2d tensors
new_x = F.pad(x, (0,0,0,n)) # pad the end of 2d tensors
new_x = F.pad(x, (0,0,0,0,0,n)) # pad the end of 3d tensors