Instruction: Target size (torch.Size([32, 9])) must be the same as input size (torch.Size([32, 10]))

Input:
I have 10 classes. I have a model such as;
from brevitas.nn import QuantLinear, QuantReLU
import torch.nn as nn

# Setting seeds for reproducibility
torch.manual_seed(0)

model = nn.Sequential(
      QuantLinear(input_size, hidden1, bias=True, weight_bit_width=weight_bit_width),
      nn.BatchNorm1d(hidden1),
      nn.Dropout(0.5),
      QuantReLU(bit_width=act_bit_width),
      QuantLinear(hidden1, hidden2, bias=True, weight_bit_width=weight_bit_width),
      nn.BatchNorm1d(hidden2),
      nn.Dropout(0.5),
      QuantReLU(bit_width=act_bit_width),
      QuantLinear(hidden2, hidden3, bias=True, weight_bit_width=weight_bit_width),
      nn.BatchNorm1d(hidden3),
      nn.Dropout(0.5),
      QuantReLU(bit_width=act_bit_width),
      QuantLinear(hidden3, num_classes, bias=True, weight_bit_width=weight_bit_width)
)

model.to(device)

and I have defined my training phase as:
def train(model, train_loader, optimizer, criterion):
    losses = []
    # ensure model is in training mode
    model.train()

    for i, data in enumerate(train_loader, 0):
        inputs, target = data['pointcloud'].to(device).float(), data['category'].to(device)
        target = torch.nn.functional.one_hot(target)
        optimizer.zero_grad()

        # forward pass
        output = model(inputs)
        loss = criterion(output, target.float())

        # backward pass + run optimizer to update weights
        loss.backward()
        optimizer.step()

        # keep track of loss value
        losses.append(loss.data.cpu().numpy())

    return losses

As I run the training code:
import numpy as np
from sklearn.metrics import accuracy_score
from tqdm import tqdm, trange

# Setting seeds for reproducibility
torch.manual_seed(0)
np.random.seed(0)

running_loss = []
running_test_acc = []
t = trange(num_epochs, desc="Training loss", leave=True)

for epoch in t:
        loss_epoch = train(model, train_loader, optimizer,criterion)
        test_acc = test(model, valid_loader)
        t.set_description("Training loss = %f test accuracy = %f" % (np.mean(loss_epoch), test_acc))
        t.refresh() # to show immediately the update
        running_loss.append(loss_epoch)
        running_test_acc.append(test_acc)

I get an error as:

Target size (torch.Size([32, 9])) must be the same as input size
(torch.Size([32, 10]))

Please help me about what can be possibly the solution. I have added one hot encoding because I have seen some solutions like that before.

Output:
The code error is pretty straightforward - the criterion (that you didn't show here in the code) expects both the input and the target arguments to be the same size, but they're not.
The problem is that you're using torch.nn.functional.one_hot(target) without telling it how many classes you need for one-hot encoding; it is then infered as the largest values in target +1 (see: https://pytorch.org/docs/stable/generated/torch.nn.functional.one_hot.html). You should change it to torch.nn.functional.one_hot(target, num_classes=10)


Instruction: modify image to black text on white background

Input:
I have an image that need to do OCR (Optical Character Recognition) to extract all data.

First I want to convert color image to black text on white background in order to improve OCR  accuracy.
I try below code
from PIL import Image
img = Image.open("data7.png")
img.convert("1").save("result.jpg")

it gave me below unclear image

I expect to have this image

Then, I will use pytesseract to get a dataframe
import pytesseract as tess
file = Image.open("data7.png")
text = tess.image_to_data(file,lang="eng",output_type='data.frame')
text

Finally,the dataframe I want to get like below

Output:
Here's a vanilla Pillow solution. Just grayscaling the image gives us okay results, but the green text is too faint.
So, we first scale the green channel up (sure, it might clip, but that's not a problem here), then grayscale, invert and auto-contrast the image.
from PIL import Image, ImageOps

img = Image.open('rqDRe.png').convert('RGB')

r, g, b = img.split()

img = Image.merge('RGB', (
    r,
    g.point(lambda i: i * 3),  # brighten green channel
    b,
))

img = ImageOps.autocontrast(ImageOps.invert(ImageOps.grayscale(img)), 5)

img.save('rqDRe_processed.png')

output


Instruction: torch.optim.LBFGS() does not change parameters

Input:
I'm trying to optimize the coordinates of the corners of an image. A similar technique works fine in Ceres Solver. But in torch.optim I'm having some issues. In particular, the optimizer for some reason does not change the parameters being optimized. I don't have much experience with pytorch, so I'm pretty sure the error is trivial. Unfortunately, reading the documentation did not help me much.
Optimization model class:
class OptimizeCorners(torch.nn.Module):
    def __init__(self, real_corners):
        super().__init__()
        self._real_corners = torch.nn.Parameter(real_corners)

    def forward(self, real_image, synt_image, synt_corners, _threshold):
        # Find homography
        if visualize_warp_interpolate:
            real_image_before_processing = real_image
            synt_image_before_processing = synt_image
        homography_matrix = kornia.geometry.homography.find_homography_dlt(synt_corners,
                                                                           self._real_corners,
                                                                           weights=None)
        # Warp and resize synt image
        synt_image = kornia.geometry.transform.warp_perspective(synt_image.float(),
                                                                homography_matrix,
                                                                dsize=(int(real_image.shape[2]),
                                                                       int(real_image.shape[3])),
                                                                mode='bilinear',
                                                                padding_mode='zeros',
                                                                align_corners=True,
                                                                fill_value=torch.zeros(3))
        # Interpolate images
        real_image = torch.nn.functional.interpolate(real_image.float(),
                                                     scale_factor=5,
                                                     mode='bicubic',
                                                     align_corners=None,
                                                     recompute_scale_factor=None,
                                                     antialias=False)
        synt_image = torch.nn.functional.interpolate(synt_image.float(),
                                                     scale_factor=5,
                                                     mode='bicubic',
                                                     align_corners=None,
                                                     recompute_scale_factor=None,
                                                     antialias=False)

        # Calculate loss
        loss_map = torch.sub(real_image, synt_image, alpha=1)

        # if element > _threshold: element = 0
        loss_map = torch.nn.Threshold(_threshold, 0)(loss_map)

        cumulative_loss = torch.sqrt(torch.sum(torch.pow(loss_map, 2)) /
                                     (loss_map.size(dim=2) * loss_map.size(dim=3)))

        return torch.autograd.Variable(cumulative_loss.data, requires_grad=True)

The way, how I am trying to execute optimization:
# Convert corresponding images to PyTorch tensors
_image = kornia.utils.image_to_tensor(_image, keepdim=False)
_synt_image = kornia.utils.image_to_tensor(_synt_image, keepdim=False)
_corners = torch.from_numpy(_corners)
_synt_corners = torch.from_numpy(_synt_corners)
# Optimizer L-BFGS
n_iters = 100
h_lbfgs = []
lr = 1
optimize_corners = OptimizeCorners(_corners)
optimizer = torch.optim.LBFGS(optimize_corners.parameters(),
                              lr=lr)
for it in tqdm(range(n_iters), desc='Fitting corners',
               leave=False, position=1):
    loss = optimize_corners(_image, _synt_image, _synt_corners, _threshold)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step(lambda: optimize_corners(_image, _synt_image, _synt_corners, _threshold))
    h_lbfgs.append(loss.item())
    print(h_lbfgs)

Output from console:
pic
So, as you can see, parameters to be optimized do not change.
UPD:
I changed return torch.autograd.Variable(cumulative_loss.data, requires_grad=True) to return cumulative_loss.requires_grad_(), and it actually works, but now I get this error after few iterations:
console output
UPD: this happens because the parameters being optimized turn into NaN after a few iterations.

Output:
After some time spent hugging the debugger, I found out that the main problem is that after a few iterations, the backward() method starts to calculate the gradient incorrectly and output NaN's. Thus, the parameters being optimized are also calclulated as NaN's. I didn't have a chance to find out exactly why this is happening, because all the traces (I used torch.autograd.set_detect_anomaly(True) method) pointed to the fact that the error occurs on the side of the C ++ Torch engine in the POW and SVD functions.
In the end, in my case, the problem was solved by the fact that I cast all parameters form float32 to float64 and reduce learning rate.
Here is the final code update can be found:
# Convert corresponding images to PyTorch tensors
            _image = kornia.utils.image_to_tensor(_image, keepdim=False).double()
            _synt_image = kornia.utils.image_to_tensor(_synt_image, keepdim=False).double()
            _corners = torch.from_numpy(_corners).double()
            _synt_corners = torch.from_numpy(_synt_corners).double()

            # Optimizer L-BFGS
            optimize_corners = OptimizeCorners(_corners)
            optimizer = torch.optim.LBFGS(optimize_corners.parameters(),
                                          max_iter=20,
                                          lr=0.01)
            torch.autograd.set_detect_anomaly(True)

            def closure():
                optimizer.zero_grad()
                loss = optimize_corners(_image, _synt_image, _synt_corners, _threshold)
                loss.backward()
                return loss

            for it in tqdm(range(100), desc="Fitting corners", leave=False, position=1):
                optimizer.step(closure)

def forward(self, real_image, synt_image, synt_corners, _threshold):
    # Find homography
    if visualize_warp_interpolate:
        real_image_before_processing = real_image
        synt_image_before_processing = synt_image

    homography_matrix = kornia.geometry.homography.find_homography_dlt(synt_corners,
                                                                       self._real_corners,
                                                                       weights=None)

    # Warp and resize synt image
    synt_image = kornia.geometry.transform.warp_perspective(synt_image,
                                                            homography_matrix,
                                                            dsize=(int(real_image.shape[2]),
                                                                   int(real_image.shape[3])),
                                                            mode='bilinear',
                                                            padding_mode='zeros',
                                                            align_corners=True,
                                                            fill_value=torch.zeros(3))

    # Interpolate images
    real_image = torch.nn.functional.interpolate(real_image,
                                                 scale_factor=10,
                                                 mode='bicubic',
                                                 align_corners=None,
                                                 recompute_scale_factor=None,
                                                 antialias=False)
    synt_image = torch.nn.functional.interpolate(synt_image,
                                                 scale_factor=10,
                                                 mode='bicubic',
                                                 align_corners=None,
                                                 recompute_scale_factor=None,
                                                 antialias=False)
    # Calculate loss
    loss_map = torch.sub(real_image, synt_image, alpha=1)

    # if element > _threshold: element = 0
    loss_map = torch.nn.Threshold(_threshold, 0)(loss_map)

    cumulative_loss = torch.sqrt(torch.sum(torch.pow(loss_map, 2)) /
                                 (loss_map.size(dim=2) * loss_map.size(dim=3)))

    return cumulative_loss.requires_grad_()


Instruction: Finding the maximum elements whose sum is less than a given value in PyTorch

Input:
Given any PyTorch 2D tensor, what will be the most efficient way to compute the number of top-K values for each row whose sum is less than a given value?
Input:
tensor([[0.6607, 0.1165, 0.0278, 0.1950],
        [0.0529, 0.4607, 0.2729, 0.2135],
        [0.3267, 0.0902, 0.4578, 0.1253]])

Required Output for the given value 0.8:
tensor([[1], #as 0.6607+0.1950 > 0.8
        [2], #as 0.4607+0.2729+0.2135 > 0.8
        [2]]) #as 0.4578+0.3267+0.1253 > 0.8

Output:
You can manage such operation by using a combination of sorting, cumulative sum, and max pooling.
First sort the values by row with torch.Tensor.sort
>>> v = x.sort(dim=1, descending=True).values
tensor([[0.6607, 0.1950, 0.1165, 0.0278],
        [0.4607, 0.2729, 0.2135, 0.0529],
        [0.4578, 0.3267, 0.1253, 0.0902]])

Then construct a mask on the cumulative sorted values that you get from applying torch.cumsum:
>>> torch.cumsum(v, dim=1) > .8
tensor([[False,  True,  True,  True],
        [False, False,  True,  True],
        [False, False,  True,  True]])

Applying a torch.Tensor.max on that mask will return the index of the first occurring True value, i.e. the location of the cumulative element which is above the threshold value 0.8:
>>> mask.max(1, True).indices
tensor([[1],
        [2],
        [2]])


Instruction: Mask the top k elements in a tensor in PyTorch (different k for each row)

Input:
For any 2D tensor X, how to get the mask for top K elements for each row where K is a tensor (not restricted to an int)?
Input:
tensor([[0.6607, 0.1165, 0.0278, 0.1950],
        [0.0529, 0.4607, 0.2729, 0.2135],
        [0.3267, 0.0902, 0.4578, 0.1253]])

Desired output: for K = torch.tensor([2,3,1])
tensor([[ True,  False, False, True],
        [ False, True,  True,  True],
        [ False, False, True, False]])

I have tried these [1], [2], but can not succeed.

Output:
You can use the torch.topk and torch.tensor.scatter_ methods for this:
K = torch.tensor([2,3,1])
for idx, k in enumerate(K):
    top_k = torch.topk(x[idx], k)
    x[idx].scatter_(0, top_k.indices, 1)
mask = x.eq(1)


Instruction: Is there a way to send location of pytorch tensor in gpu memory between docker containers and build them in different containers

Input:
To quickly sum up the problem, I need to transfer images (size is (1920,1200,3)) between PyTorch docker containers and process them. Containers are located in the same system. Speed is very important and transfer should not take more than 2-3ms one way. Two containers will be shared via IPC so I find no problem transferring NumPy arrays via shared memory using buffers (example https://docs.python.org/3/library/multiprocessing.shared_memory.html). I am curious is there a similar way to do that with PyTorch tensors allocated on GPU?
From what I've learned, CUDA Tensors are already in the shared memory. I tried transferring them and Pytorch Tensor Storage objects via socket but it takes around 50-60ms one way, which is way too slow. For testing purposes, I just run 2 programs in separate terminals.
Container 1 code:
import torch
import zmq

def main():
    ctx = zmq.Context()
    sock = ctx.socket(zmq.REQ)
    sock.connect('tcp://0.0.0.0:6000')

    x = torch.randn((1, 1920, 1200, 3), device='cuda')
    storage = x.storage()
    while True:
        sock.send_pyobj(storage)
        sock.recv()

if __name__ == "__main__":
    main()

Container 2 code:

import torch
import zmq
import time

def main():
    ctx = zmq.Context()
    sock = ctx.socket(zmq.REP)
    sock.bind('tcp://*:6000')

    for i in range(10):
        before = time.time()
        storage = sock.recv_pyobj()
        tensor = torch.tensor((), device=storage.device)
        tensor.set_(storage)
        after = time.time()
        print(after - before)
        sock.send_string('')

if __name__ == "__main__":
    main()

Edit:
I found a similar topic discussed 4 years ago. There person extracts additional information from storage using share_cuda() function, which gives cudaIpcMemHandle_t.
Is there a way to reconstruct Storage/Tensor using cudaIpcMemHandle_t or information extracted from share_cuda() function using Pytoch functional? or there is a better way to achieve the same result?

Output:
I found a function in torch.multiprocessing.reductions that rebuilds tensors from the output generated by _share_cuda_(). Now my code looks something like this:
Container 1 code:
import torch
import zmq

def main():
    ctx = zmq.Context()
    sock = ctx.socket(zmq.REQ)
    sock.connect('tcp://0.0.0.0:6000')

    image = torch.randn((1, 1920, 1200, 3), dtype=torch.float, device='cuda:0')
    storage = image.storage()

    (storage_device, storage_handle, storage_size_bytes, storage_offset_bytes,
    ref_counter_handle, ref_counter_offset, event_handle, event_sync_required) = storage._share_cuda_()

    while True:
        sock.send_pyobj({
            "dtype": image.dtype,
            "tensor_size": (1920, 1200, 3),
            "tensor_stride": image.stride(),
            "tensor_offset": image.storage_offset(), # !Not sure about this one.
            "storage_cls": type(storage),
            "storage_device": storage_device,
            "storage_handle": storage_handle,
            "storage_size_bytes": storage_size_bytes,
            "storage_offset_bytes":storage_offset_bytes,
            "requires_grad": False,
            "ref_counter_handle": ref_counter_handle,
            "ref_counter_offset": ref_counter_offset,
            "event_handle": event_handle,
            "event_sync_required": event_sync_required,
        })

        sock.recv_string()

if __name__ == "__main__":
    main()

Container 2 code:
import torch
import zmq
import time
from torch.multiprocessing.reductions import rebuild_cuda_tensor

def main():
    ctx = zmq.Context()
    sock = ctx.socket(zmq.REP)
    sock.bind('tcp://*:6000')

    for i in range(10):
        before = time.time()

        cuda_tensor_info = sock.recv_pyobj()
        rebuilt_tensor = rebuild_cuda_tensor(torch.Tensor, **cuda_tensor_info)

        after = time.time()
        print(after - before)

        sock.send_string('')

if __name__ == "__main__":
    main()


Instruction: transforms.ToTensor() and Numpy Issue

Input:
I'm working on MNIST datasets using Pytorch and I'm trying to scale the images, I ran into problems associated with Numpy
train_dataset = datasets.MNIST(root='data',
                                train=True,
                                transform=transforms.ToTensor(),
                                download=True)

And here's my error, really confused on how to solve this
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-ef2899ce3492> in <module>
      6                                 train=True,
      7                                 transform=transforms.ToTensor(),
----> 8                                 download=True)
      9
     10 test_dataset = datasets.MNIST(root='data',

~\miniconda3\envs\6.86x\lib\site-packages\torchvision\datasets\mnist.py in __init__(self, root, train, transform, target_transform, download)
     91                                ' You can use download=True to download it')
     92
---> 93         self.data, self.targets = self._load_data()
     94
     95     def _check_legacy_exist(self):

~\miniconda3\envs\6.86x\lib\site-packages\torchvision\datasets\mnist.py in _load_data(self)
    110     def _load_data(self):
    111         image_file = f"{'train' if self.train else 't10k'}-images-idx3-ubyte"
--> 112         data = read_image_file(os.path.join(self.raw_folder, image_file))
    113
    114         label_file = f"{'train' if self.train else 't10k'}-labels-idx1-ubyte"

~\miniconda3\envs\6.86x\lib\site-packages\torchvision\datasets\mnist.py in read_image_file(path)
    507
    508 def read_image_file(path: str) -> torch.Tensor:
--> 509     x = read_sn3_pascalvincent_tensor(path, strict=False)
    510     assert(x.dtype == torch.uint8)
    511     assert(x.ndimension() == 3)

~\miniconda3\envs\6.86x\lib\site-packages\torchvision\datasets\mnist.py in read_sn3_pascalvincent_tensor(path, strict)
    496     parsed = np.frombuffer(data, dtype=m[1], offset=(4 * (nd + 1)))
    497     assert parsed.shape[0] == np.prod(s) or not strict
--> 498     return torch.from_numpy(parsed.astype(m[2])).view(*s)
    499
    500

RuntimeError: Numpy is not available

Output: Figured it out, my Numpy version was outdated compared to pytorch. Updated it basically!


Instruction: Does PyTorch allocate GPU memory eagerly?

Input:
Consider the following script:
import torch

def unnecessary_compute():
    x = torch.randn(1000,1000, device='cuda')
    l = []
    for i in range(5):
        print(i,torch.cuda.memory_allocated())
        l.append(x**i)
unnecessary_compute()

Running this script with PyTorch (1.11) generates the following output:
0 4000256
1 8000512
2 12000768
3 16001024
4 20971520

Given that PyTorch uses asynchronous computation and we never evaluated the contents of l or of a tensor that depends on l, why did PyTorch eagerly allocate GPU memory to the new tensors? Is there a way of invoking these tensors in an utterly lazy way (i.e., without triggering GPU memory allocation before it is required)?

Output:
torch.cuda.memory_allocated() returns the memory that has been allocated, not the memory that has been "used".
In a typical GPU compute pipeline, you would record operations in a queue along with whatever synchronization primitives your API offers. The GPU will then dequeue and execute those operations, respecting the enqueued synchronization primitives. However, GPU memory allocation is not usually an operation which even goes on the queue. Rather, there's usually some sort of fundamental instruction that the CPU can issue to the GPU in order to allocate memory, just as recording operations is another fundamental instruction. This means that the memory necessary for a GPU operation has to be allocated before the operation has even been enqueued; there is no "allocate memory" operation in the queue to synchronize with.
Consider Vulkan as a simple example. Rendering operations are enqueued on a graphics queue. However, memory is typically allocated via calls to vkAllocateMemory(), which does not accept any sort of queue at all; it only accepts the device handle and information about the allocation (size, memory type, etc). From my understanding, the allocation is done "immediately" / synchronously (the memory is safe to use by the time the function call returns on the CPU).
I don't know enough about GPUs to explain why this is the case, but I'm sure there's a good reason. And perhaps the limitations vary from device to device. But if I were to guess, memory allocation probably has to be a fairly centralized operation; it can't be done by just any core executing recorded operations on a queue. This would make sense, at least; the space of GPU memory is usually shared across cores.
Let's apply this knowledge to answer your question: When you call l.append(x**i), you're trying to record a compute operation. That operation will require memory to store the result, and so PyTorch is likely allocating the memory prior to enqueuing the operation. This explains the behavior you're seeing.
However, this doesn't invalidate PyTorch's claims about asynchronous compute. The memory might be allocated synchronously, but it won't be populated with the result of the operation until the operation has been dequeued and completed by the GPU, which indeed happens asynchronously.


Instruction: How to use transforms.FiveCrop() outside the DataLoader?

Input:
I have a dataloader that returns a batch of shape torch.Size([bs, c, h, w]) where bs=4, c=1,and (h, w=128). Now I want to apply some custom transformations to the returned batch. Note that I can not apply transformations in the Dataloader as I need to feed the returned batch as is to one network and a transformed one to another network.
More specifically, I want to apply the following transformations to the returned batch:
1. CenterCrop(100)
2. FiveCrop(16)
3. Resize(128)
4. ToTensor()
5. Normalize([0.5], [0.5])

I have created a function to achieve the following task as follows:
# DataLoader code
#
#
orig_img = next(iter(DataLoader))
patches = get_patches(orig_img)

def get_patches(orig_img):
    # orig_img.shape = torch.Size([4, 1, 128, 128])
    images = [TF.to_pil_image(x) for x in orig_img.cpu()]
    resized_imgs = []

    for img in images:
        img = transforms.CenterCrop(100)(img)
        five_crop = transforms.FiveCrop(64)(img)
        f_crops = transforms.Lambda(lambda crops: torch.stack([transforms.Normalize([0.5], [0.5])(transforms.ToTensor()(transforms.Resize(128)(crop))) for crop in crops]))(five_crop)
        resized_imgs.append(f_crops)
    return resized_imgs

The problem right now is that when I get the resized_imgs list, every tensor inside it looses the batch size dimension i.e. resized_imgs[0].shape =  torch.Size([ncrops, c, h, w]) (4d) whereas, I expect the shape to be torch.Size([bs, ncrops, c, h, w]) (5d).

Output:
Your data loader will return a tensor of shape (bs, c, h, w). Therefore orig_img is shaped the same way and iterating through it will provide you with a tensor img shaped as (c, h, w). Applying FiveCrop will create an additional dimension such that five_crop is shaped (5, c, h, w). Then f_crops will be shaped (5, c, 128, 128). Finally, the tensor is appended with the others in resized_imgs (the list containing the different patched images). All in all resized_imgs contains bs elements since orig_img.size(0) = bs, and each element is a tensor shaped (5, c, 128, 128) (five patches per image) as we've described above.
Another way of writing this function would be:
def get_patches(orig_img):
    # orig_img.shape = (4, 1, 128, 128)

    img_t = T.Compose([T.ToPILImage(),
                       T.CenterCrop(100),
                       T.FiveCrop(64)])
    patch_t = T.Compose([T.Resize(128),
                         T.ToTensor(),
                         T.Normalize([0.5], [0.5])])

    resized_imgs = []

    for img in orig_img:
        five_crop = img_t(img)
        f_crops = torch.stack(list(map(patch_t, five_crop)))
        resized_imgs.append(f_crops)
    return torch.stack(resized_imgs)

The last line will stack all image patches into a single tensor of shape (bs, 5, c, 128, 128).


Instruction: constant loss and accuracy in Pytorch model

Input:
I have created a binary classification model. During training the model I am getting both loss and accuracy as constant. I tried the same code for other datasets and it is giving the same thing in every case. How to fix this?
Here is the code
model=Model()
lossfn=nn.BCEWithLogitsLoss()
optimizer=torch.optim.Adam(model.parameters(),lr=0.01)
#training model
epochs=10
train_acc=[]
test_acc=[]
losses=torch.zeros(epochs)

#training over loops
for i in range(epochs):
    model.train()
    batchAcc=[]
    batchLoss=[]

    for X,y in train_loader:

        #forward pass
        yHat=model(X)

        #lossfunction
        loss=lossfn(y,yHat)

        #backprop
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        batchLoss.append(loss.item())
        # predictions=(torch.sigmoid(yHat)>0.5).float()
        predictions=yHat>0
        batchLoss.append(loss.item())
        batchAcc.append(100*torch.mean(((yHat>0) == y).float()))

    losses[i]=np.mean(batchLoss)
    train_acc.append(np.mean(batchAcc))

    #Evaluation mode
    model.eval()
    X,y=next(iter(test_loader))
    with torch.no_grad():
        yHat=model(X)
    test_acc.append( 100*torch.mean(((yHat>0) == y).float()))

    print(f'Epoch:{i+1} loss:{loss.item()} train accuracy:{train_acc[-1]} test accuracy:{test_acc[-1]}')

Output of the code
Epoch:1 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:2 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:3 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:4 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:5 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:6 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:7 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:8 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:9 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0
Epoch:10 loss:0.6931471824645996 train accuracy:33.33646011352539 test accuracy:0.0

Output:
The other answers are wrong, the order of
optimizer.zero_grad() # Clears the gradient for all parameters
loss.backward() # Populates the gradient for all parameters
optimizer.step() # Uses the gradient to update the parameters

is correct!
Likely culprit is this line:
loss=lossfn(y,yHat)

In your case, y is the ground truth and yHat is the prediction (which is confusing in the usual convention, hence the mistake).
However, PyTorch losses expect the prediction first and the target as the second argument; the gradient is not propagated correctly through the targets. Therefore, you should change it to
loss=lossfn(yHat,y)

Or even better, call yHat the ground truth target and y the prediction.


Instruction: Is there a way to specify the output dimension of pytorch least square solution?

Input:
With 3 by n by k tensor A and 1 by k by m tensor x we can haveAx = B where B has shape of [3, n, m]
torch.linalg.lstsq(A, B) returns a 3 x k x mtensor as solution. Is there a way to find the 1 by k by m tensor x?

Output:
The difference between torch.lingalg.lstsq and torch.matmul is that torch.lingalg.lstsq computes its answer based on batch-wise operation while torch.matmul does not.
And your 1 by k by m solution will be non-batch wise solution or some kind of global solution that can commonly be applied across whole batch. This case, you can simply reduce the batch dimension and obtain your least square solution.
A_re = A.reshape(1,-1, k)
B_re = B.reshape(1, -1, m) # or torch.matmul(A_re, x)
x = torch.linalg.lstsq(A_re, B_re)
x.size()
> torch.Size([1, k, m])


Instruction: Pytorch matrix size issue does not multiply

Input:
This may have been answered before so happy about any links. I am new to pytorch and do not understand why my Conv2d pipeline is failing with
mat1 and mat2 shapes cannot be multiplied (64x49 and 3136x512)

        self.net = nn.Sequential(
            nn.Conv2d(in_channels=c, out_channels=32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=2),
            nn.ReLU(),
            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.Flatten(),
            nn.Linear(3136, 512),
            nn.ReLU(),
            nn.Linear(512, output_size)
        )

with input shape 1x84x84.
I did the calculation and this is the size that breaks down over the different steps with the kernel and size settings per layer.
84 ->  K:8 , S:4 => 20
20 -> K:3 , S:2 => 9
9 -> K:3 , S:1 => 7

7^2 * 64 => 3136  for the flattened layer

I am not sure where the 64x49 is coming from .

Output:
I have tried your model and your calculation is totally correct. The problem lies in your input. For torch calculation, if your input shape is 1x84x84, a 3d torch, you should input a 4d torch indeed, where the first dimension represent the batch-size. You may find more information about batch, which is widely used to enhance computation speed.
If you just want to test on single data, you may just add a dimension like x = x[None, :] to make it become 4d torch. This will be a quick fix to your problem


Instruction: Is it necessary to use with torch.no_grad() for feature extraction?

Input:
I'm attempting feature extraction in an unorthodox way. I extract features in eval() mode to switch off the batch norm and dropout layers and use the running means and std provided by ImageNet.
I use a feature extractor to extract features from two related images and concatenate the two tensors stackwise before passing through a linear dense classifier model for training. I'm wondering whether I can avoid using  with torch.no_grad() as the two models are unrelated.
Here is a simplified version:
num_classes = 2
num_epochs = 10
criterion = nn.CrossEntropyLoss().to(device)
optimizer = torch.optim.Adam(classifier.parameters(), lr=0.001)

densenet= DenseNetConv()

# set densenet to eval to switch off batch norm and dropout layers and use ImageNet running means/ std devs
densenet.eval()
densenet.to(device)

classifier = nn.Linear(4416, num_classes)
classifier.to(device)

for epoch in range(num_epochs):

  classifier.train()

  for i, (inputs_1, inputs_2,  labels) in enumerate(dataloaders_dict['train']):
       inputs_1= inputs_1.to(device)
       inputs_2 = inputs_2.to(device)
       labels = labels.to(device)

       features_1 = densenet(inputs_1) # extract features 1
       features_2 = densenet(inputs_2) # extract features 2

       combined = torch.cat([features_1, features_2], dim=1) # combine features
       combined = combined(-1, 4416) # reshape

       optimizer.zero_grad()

       # Forward pass to get output/logits
       outputs =  classifier(combined)

       # Calculate Loss: softmax --> cross entropy loss
       loss = criterion(outputs, labels)

       _, pred = torch.max(outputs, 1)
       equality_check = (labels.data == pred)

       # Getting gradients w.r.t. parameters
       loss.backward()
       optimizer.step()

As you can see, I do not call  with torch.no_grad(), despite having densenet.eval() as my separate feature extractor. Is there an issue with the way this is implemented or can I assume that this will not interfere with the classifier model?

Output:
If you are doing inference on a model, applying torch.no_grad() won't have any effect on the resulting output. As you've said only nn.Module.eval will since it modifies how the forward operation is performed (namely which statistics to use to normalize the batch elements).
It is recommended to switch off gradient computation when backpropagation is not necessary. This avoids caching activations on forward call resulting in faster inference time.

In your case, you can either wrap your inference call on densenet with torch.no_grad:
torch.no_grad():
    features_1 = densenet(inputs_1) # extract features 1
    features_2 = densenet(inputs_2) # extract features 2

Or alternatively, switch off the requires_grad flag on your module's parameter tensors using nn.Module.requires_grad_:
densenet.eval()
densenet.requires_grad_(False)


Instruction: (Pytorch) Why conv2d results are all different. Their data type is all integer, no float

Input:
I've tried and compared three different methods for convolution computation with a custom kernel in Pytorch. Their results are different but I don't understand why that is.
Setup code:
import torch
import torch.nn.functional as F

inp = torch.arange(3*500*700).reshape(1,3,500,700).to(dtype=torch.float32)
wgt = torch.ones((1,3,3,3)).to(dtype=torch.float32)
stride = 1
padding = 0
h = inp.shape[2] - wgt.shape[2] + 1
w = inp.shape[3] - wgt.shape[3] + 1

Method 1
out1 = torch.zeros((1,h,w)).to(dtype=torch.float32)
for o in range(1):
    for i in range(3):
        for j in range(h):
            for k in range(w):
                out1[o,j,k] = out1[o,j,k] + (inp[0, i, j*stride:j*stride+3, k*stride:k*stride+3] * wgt[0,i]).sum()
out1 = out1.to(dtype=torch.int)

Method 2
inp_unf = F.unfold(inp, (3,3))
out_unf = inp_unf.transpose(1,2).matmul(wgt.view(1,-1).t()).transpose(1,2)
out2 = F.fold(out_unf, (h,w), (1,1))
out2 = out2.to(dtype=torch.int)

Method 3
out3 = F.conv2d(inp, wgt, bias=None, stride=1, padding=0)
out3 = out3.to(dtype=torch.int)

And here are the results comparison:
>>> h*w
347604

>>> (out1==out2).sum().item()
327338
>>> (out2 == out3).sum().item()
344026
>>> (out1 == out3).sum().item()
330797

>>> out1.shape
(1, 498, 698)
>>> out2.shape
(1, 1, 498, 698)
>>> out3.shape
(1, 1, 498, 698)

Their data types are all int so floating point won't the result. When I use a squared input format such as h=500 and w=500, all three results are all matching. But not for non-squared inputs, such as the one above with h=500 and w=700. Any insight?

Output:
All three results are cast to integer data types but keep in mind their computation is done with float32... It is often preferred to check equality between two tensors using torch.isclose:
>>> torch.isclose(out1, out2).float().mean()
tensor(1.)

>>> torch.isclose(out2, out3).float().mean()
tensor(1.)

>>> torch.isclose(out1, out3).float().mean()
tensor(1.)


Instruction: Pytorch: How to prepare 1d dataset from pandas dataframe?

Input:
I am trying to make 1d Dataset from a pandas data frame, however, the output is weird.
I wrote the code to convert dataset from pandas dataframe: size is 8000x512,
# create dataset
class carte_dataset(Dataset):
    def __init__(self,root):
        self.root = root
        self.df = pd.read_csv(root,index_col=0)
        self.X = torch.tensor(self.df.iloc[:,1:].values)
        self.regi_no =  self.df.iloc[:,0].values

    def __len__(self):
        return len(self.regi_no)

    def __getitem__(self,idx):
        return self.X[idx],self.regi_no[idx]

Then, I confirmed the tensor size
dataset = carte_dataset(root)
data,_ = dataset.__getitem__(0)
data.size()

I expected the size was torch.Size([1,512]), but the shape was torch.Size([512]).
Is the way to make 1d dataset from the pandas dataframe appropriate? Also, if this way is incorrect, how I should revise this code?

Output:
What you need to do is to wrap the dataset with the dataloader which will have the effect of

retrieving the individual element tuple pairs from the underlying dataset: self.X[idx], self.regi_no[idx], shaped (512,) and (1,) respectively.

and collating them to form two batches of input/labels shaped (bs, 512) and bs, 1) where bs is the batch size.

The standard dataloader utility in PyTorch is torch.utils.data.DataLoader:
>>> dataloader = data.DataLoader(dataset, batch_size=1, shuffle=False)

Then you can iterate through the dataset via the dataloader:
>>> for x, y in dataloader:
...     # x shaped (1, 512), corresponds to [X[0]]
...     # y shaped (1, 1), corresponds to [regi_no[0]]


Instruction: Pytorch, get the index of the first 0 in as mask?

Input:
I have a tensor that looks like: (1, 1, 1, 1, 1, 1, 1, 1, 0, 0).
I want to get the index where the first zero appears.
What would the be best way do this?

Output:
Not the best usage of argmin but it should work here I think:
>>> torch.tensor([1, 1, 1, 1, 1, 1, 1, 1, 0, 0]).argmin()
tensor(8)


Instruction: Pytorch: Set indexes in a tensor based on a list of tensor indices

Input:
Is there a way to efficiently set the values of a tensor based on a tensor of indices and a tensor of values?
tensor_to_change = tensor([[-36.9127, -45.6596, -47.1595],
        [-36.9409, -45.7024, -47.2050],
        [-36.9865, -45.7665, -47.2711],
        [-36.3202, -36.9561, -47.2066],
        [-36.2929, -36.9333, -47.1702]]
tensor_of_indices = tensor([[0],
        [0],
        [0],
        [1],
        [1]])
tensor_of_values = tensor([[-37.9409],
        [-38.4865],
        [-36.9561],
        [-34.9561],
        [-38.7562]])

I can accomplish this in with a for loop, but this step then becomes really slow:
for i, a in enumerate(tensor_of_indices):
    tensor_to_change[i][a] = tensor_of_values[i]

Is there a torch function which can do this faster?

Output:
Try this:
rows = torch.arange(tensor_to_change.size(0))
cols = tensor_of_indices.squeeze()
tensor_to_change[rows, cols] = tensor_of_values.squeeze()

Output:
tensor_to_change
>tensor([[-37.9409, -45.6596, -47.1595],
        [-38.4865, -45.7024, -47.2050],
        [-36.9561, -45.7665, -47.2711],
        [-36.3202, -34.9561, -47.2066],
        [-36.2929, -38.7562, -47.1702]])


Instruction: When training with pytorch, debugger hangs, even though running works fine

Input:
Trying to train with pytorch hangs in debug mode, but works in run mode.
    sampler_train = WeightedRandomSampler(
        sample_weights_train,
        num_samples=len(sample_weights_train),
        replacement=True
    )

    train_loader = torch.utils.data.DataLoader(
        train_set,
        sampler=sampler_train,
        batch_size=32,
        num_workers=2
    )

   for epoch in range(10):
        for i, data in enumerate(train_loader, 0):
            model.train()
            print("something")

After placing a breakpoint on model.train(), then moving on to the next line, "something" is never printed in debug mode, but is printed in run mode in Pycharm.
How to debug my code?

Output:
After a long search, I found the answer here, which led to here

Setting Gevent Compatible in Preferences | Build, Execution,
Deployment | Python Debugger solves the issue.


Instruction: pytorch batchwise indexing

Input:
I am searching for a way to do some batchwise indexing for tensors.
If I have a variable Q of size 1000, I can get the elements I want by
Q[index], where index is a vector of the wanted elements.
Now I would like to do the same for more dimensional tensors.
So suppose Q is of shape n x m and I have a index matrix of shape n x p.
My goal is to get for each of the n rows the specific p elements out of the m elements.
But Q[index] is not working for this situation.
Do you have any thoughts how to handle this?

Output:
You can seem to be a simple application of torch.gather which doesn't require any additional reshaping of the data or index tensor:
>>> Q = torch.rand(5, 4)
tensor([[0.8462, 0.3064, 0.2549, 0.2149],
        [0.6801, 0.5483, 0.5522, 0.6852],
        [0.1587, 0.4144, 0.8843, 0.6108],
        [0.5265, 0.8269, 0.8417, 0.6623],
        [0.8549, 0.6437, 0.4282, 0.2792]])

>>> index
tensor([[0, 1, 2],
        [2, 3, 1],
        [0, 1, 2],
        [2, 2, 2],
        [1, 1, 2]])

The following gather operation applied on dim=1 return a tensor out, such that:
out[i, j] = Q[i, index[i,j]]

This is done with the following call of torch.Tensor.gather on Q:
>>> Q.gather(dim=1, index=index)
tensor([[0.8462, 0.3064, 0.2549],
        [0.5522, 0.6852, 0.5483],
        [0.1587, 0.4144, 0.8843],
        [0.8417, 0.8417, 0.8417],
        [0.6437, 0.6437, 0.4282]])


Instruction: Does using queue in a multiprocessing Process use pickling?

Input:
Consider the following example from Python documentation. Does multiprocessing.Process use serialization (pickle) to put items in the shared queue?
from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print(q.get())    # prints "[42, None, 'hello']"
    p.join()

I understand multiprocessing.Process uses pickle to serialize / deserialize data to communicate with the main process. And threading.Thread does not need serialization, so it does not use pickle. But I'm not sure how communication with Queue happens in a multiprocessing.Process.
Additional Context
I want multiple workers to fetch data from a database (or local storage) to fill the shared queue where the items are consumed by the main process sequentially. Each record that is fetched is large (1-1.5 mb). The problem with using multiprocessing.Process is that serialization / deserialization of the data takes a long time. Pytorch's DataLoader makes use of this and is therefore unsuitable for my use case.
Is multi-threading the best alternative for such a use case?

Output:
Yes, mutiprocessing's queues does use Pickle internally. This can be seen in multiprocessing/queues.py of the CPython implementation. In fact, AFAIK CPython uses Pickle for transferring any object between interpreter processes. The only way to avoid this is to use shared memory but it introduces strong limitation and cannot basically be used for any type of objects.
Multithreading is limited by the Global Interpreter Lock (GIL) which basically prevent any parallel speed up except of operations releasing the GIL (eg. some Numpy functions) and IO-based ones.
Python (and especially CPython) is not the best languages for parallel computing (nor for high performance). It has not been designed with that in mind and this is nowadays a pretty strong limitation regarding the recent sharp increase of the number of cores per processor.


Instruction: Reading .pyth filetype in PyTorch

Input:
There is pretrained model in a repository that its file type is .pyth. I searched the web to find out about this file type and which language is able to read that but I could not find anything. Since I am working with PyTorch, is it possible to read such file in PyTorch? Moreover, normally how it is possible  to read and generate that?
To be clearer, in the repository of the TimeSformer model, the pretrained models are of this filetype and as an example, you can find the following commands in that repository:
import torch
from timesformer.models.vit import TimeSformer

model = TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',  pretrained_model='/path/to/pretrained/model.pyth')

dummy_video = torch.randn(2, 3, 8, 224, 224) # (batch x channels x frames x height x width)

pred = model(dummy_video,) # (2, 400)

Output:
The file extension can literally be anything, it doesn't change the file contents. If you run torch.load("file.pyth") it will load a weight dictionary. You can find this in the code in the repo you included. They save the model using this code:
path_to_checkpoint = get_path_to_checkpoint(path_to_job, epoch + 1)
with PathManager.open(path_to_checkpoint, "wb") as f:
    torch.save(checkpoint, f)

and the get_path_to_checkpoint function can be found here:
def get_path_to_checkpoint(path_to_job, epoch):
    """
    Get the full path to a checkpoint file.
    Args:
        path_to_job (string): the path to the folder of the current job.
        epoch (int): the number of epoch for the checkpoint.
    """
    name = "checkpoint_epoch_{:05d}.pyth".format(epoch)
    return os.path.join(get_checkpoint_dir(path_to_job), name)

So, they just pass a filename with an extension .pyth to torch.save.


Instruction: How to bound the output of a layer in pytorch

Input:
I want my model to output a single value, how can I constrain the value to (a, b)?
for example, my code is:
class ActorCritic(nn.Module):
    def __init__(self, num_state_features):
        super(ActorCritic, self).__init__()

        # value
        self.critic_net = nn.Sequential(
            nn.Linear(num_state_features, 64),
            nn.ReLU(),
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 1)
        )

        # policy
        self.actor_net = nn.Sequential(
            nn.Linear(num_state_features, 64),
            nn.ReLU(),
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 1),
        )

    def forward(self, state):
        value = self.critic_net(state)
        policy_mean = self.actor_net(state)
        return value, policy_mean

and I want the policy output to be in the range (500, 3000), how can I do this?
(I have tried torch.clamp(), this does not work well since the policy would stay always the same if it is near the limit, for example the output goes to -1000000 and it will then stay 500 forever, or takes really long time to change. The same is true for function like nn.Sigmoid())

Output:
Use an activation function on the final layer that bounds the outputs in some range, then normalize to your desired range. For instance, sigmoid function bound the output in the range [0,1].
output = torch.sigmoid(previous_layer_output) # in range [0,1]
output_normalized = output*(b-a) + a          # in range [a,b]


Instruction: Cannot import Pytorch Dataset class written in another py file

Input:
I am trying to make my main.py file concise so I wrote my Dataset class in another py file.
from torch.utils.data import DataLoader, Dataset
from sklearn.model_selection import train_test_split
from torch.nn.utils.rnn import pad_sequence

class mydata(Dataset):
   def __init__(self, X, y):
        self.X = torch.FloatTensor(X)
        self.y = torch.FloatTensor(y)

   def __len__(self):
        return len(self.X)

   def __getitem__(self, index):
        y = self.y[index]
        X = self.X[index]
        return X,

When I try to import it, I got an import error.

ImportError: cannot import name 'mydata' from 'tools' (d:\A\Pycodes\tools.py)```

Output: The problem is solved as tools.py is restricted and probably a function already existed in the environment.


Instruction: Does model.eval() placement matter in the code?

Input:
Might be a bit silly but I need to make sure this is correct. Does it matter if I place my code like this?:
model.eval()
with torch.no_grad():

Or can I get the same behaviour like this:
with torch.no_grad():
    model.eval()

I'm just wondering because I have a function that has model.eval() inside of it which goes inside of a loop where with torch.no_grad(): is before it ...

Output:
both of them are correct, you just need to use the model. eval() Before you explore,
you should put the model in eval mode, both in general and so that batch norm
doesn't cause you issues and is using its eval statistics


Instruction: Conda is installing a very old version of pytorch-lightning

Input:
I tried installing pytorch lightning by runnning:
conda install -c conda-forge pytorch-lightning

as described here: https://anaconda.org/conda-forge/pytorch-lightning
This link seems updated to version 1.6.5
However, when I run this command, an old version of pytorch-lightning is installed, as can be seen here:

> The following NEW packages will be INSTALLED:
>
>   absl-py            pkgs/main/noarch::absl-py-0.15.0-pyhd3eb1b0_0
> blinker            pkgs/main/linux-64::blinker-1.4-py39h06a4308_0
> google-auth-oauth~ pkgs/main/noarch::google-auth-oauthlib-0.4.1-py_2
> oauthlib           pkgs/main/noarch::oauthlib-3.2.0-pyhd3eb1b0_0
> pytorch-lightning  conda-forge/noarch::pytorch-lightning-0.8.5-py_0
> requests-oauthlib  pkgs/main/noarch::requests-oauthlib-1.3.0-py_0
> tensorboard        pkgs/main/noarch::tensorboard-2.6.0-py_1
> tensorboard-data-~
> pkgs/main/linux-64::tensorboard-data-server-0.6.0-py39hca6d32c_0
> tensorboard-plugi~ pkgs/main/noarch::tensorboard-plugin-wit-1.6.0-py_0

As you can see, version 0.8.5 is being installed. Is there a way for me to use conda and get a newer version of pytorch-lightning?
Things I have tried:

updating conda
using both linux and windows

Output: The problem was solved by launching a new conda environment. Apparently there were some conflicting pre installed dependencies.


Instruction: CUDA error: invalid device ordinal when using python 3.9

Input:
I'm trying to excute a code, but I keep geting this error when compiling this piece of code:
  import tensorflow as tf

  from xba import XBA

  import torch

  torch.tensor([1, 2, 3, 4]).to(device="cuda:2")

torch.tensor([1, 2, 3, 4]).to(device="cuda:2") generates this error: "
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the
stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

AnyIdea about the origine of the bug, it's just the first line of code!!

Output:
"cuda:2"

selects the third GPU in your system.  If you don't have 3 GPUs (at least) in your system, you'll get this error.
Assuming you have at least 1 properly installed and set up CUDA GPU available, try:
"cuda:0"


Instruction: Normalized Cross Entropy Loss Implementation Tensorflow/Keras

Input:
I am trying to implement a normalized cross entropy loss as described in this publication
The math given is:

This paper provided a PyTorch implementation:
@mlconfig.register
class NormalizedCrossEntropy(torch.nn.Module):
    def __init__(self, num_classes, scale=1.0):
        super(NormalizedCrossEntropy, self).__init__()
        self.device = device
        self.num_classes = num_classes
        self.scale = scale
    def forward(self, pred, labels):
        pred = F.log_softmax(pred, dim=1)
        label_one_hot = torch.nn.functional.one_hot(labels, self.num_classes).float().to(self.device)
        nce = -1 * torch.sum(label_one_hot * pred, dim=1) / (- pred.sum(dim=1))
        return self.scale * nce.mean()

But I need this to be translated to tensorflow for my ongoing project. Can anyone help me implement this normalized crossentropy loss in tensorflow?

Output:
I think is just a matter of translating methods name:
# given y_pred as 1-hot and y-true the multiclass probabilities
def NCE(y_true, y_pred):
     num = - tf.math.reduce_sum(tf.multiply(y_true, y_pred), axis=1)
     denom = -tf.math.reduce_sum(y_pred, axis=1)
     return tf.reduce_mean(num / denom)
t = tf.constant([[1,0,0], [0,0,1]], dtype=tf.float64)
y = tf.constant([[0.3,0.6,0.1], [0.1,0.1,0.8]], dtype=tf.float64)
NCE(t,y)
# <tf.Tensor: shape=(), dtype=float64, numpy=0.55>

Just check if the resulting loss is the same since I've not tested it


Instruction: Loading a modified pretrained model using strict=False in PyTorch

Input:
I want to use a pretrained model as the encoder part in my model. You can find a version of my model:
class MyClass(nn.Module):
    def __init__(self, pretrained=False):
        super(MyClass, self).__init__()
        self.encoder=S3D_featureExtractor_multi_output()
        if pretrained:

        weight_dict=torch.load(os.path.join('models','weights.pt'))

        model_dict=self.encoder.state_dict()

        list_weight_dict=list(weight_dict.items())
        list_model_dict=list(model_dict.items())

        for i in range(len(list_model_dict)):
            assert list_model_dict[i][1].shape==list_weight_dict[i][1].shape
            model_dict[list_model_dict[i][0]].copy_(weight_dict[list_weight_dict[i][0]])

        for i in range(len(list_model_dict)):
            assert torch.all(torch.eq(model_dict[list_model_dict[i][0]],weight_dict[list_weight_dict[i][0]].to('cpu')))
        print('Loading finished!')

def forward(self, x):

    a, b = self.encoder(x)
    return a, b

Because I modified some parts of the code of this pretrained model, based on this post I need to apply strict=False to avoid facing error, but based on the scenario that I load the pretrained weights, I cannot find a place in the code to apply strict=False. How can I apply that or how can I change the scenario of loading the pretrained model taht makes it possible to apply strict=False?

Output:
strict = False is to specify when you use load_state_dict() method. state_dict are just Python dictionaries that helps you save and load model weights.
(for more details, see https://pytorch.org/tutorials/recipes/recipes/what_is_state_dict.html)
If you use strict=False in load_state_dict, you inform PyTorch that the target model and the original model are not identical, so it just initialises the weights of layers which are present in both and ignores the rest.
(see https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=load_state_dict#torch.nn.Module.load_state_dict)
So, you will need to specify the strict argument when you load the pretrained model weights. load_state_dict can be called at this step.
If the model for which weights must be loaded is self.encoder
and if state_dict can be retrieved from the model you just loaded, you can just do this
loaded_weights = torch.load(os.path.join('models','weights.pt'))
self.encoder.load_state_dict(loaded_weights, strict=False)

for more details and a tutorial, see https://pytorch.org/tutorials/beginner/saving_loading_models.html .


Instruction: Accessing a specific layer in a pretrained model in PyTorch

Input:
I want to extract the features from certain blocks of the TimeSformer model and also want to remove the last two layers.
import torch
from timesformer.models.vit import TimeSformer

model = TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',  pretrained_model='/path/to/pretrained/model.pyth')

The print of the model is as follows:
TimeSformer(
  (model): VisionTransformer(
(dropout): Dropout(p=0.0, inplace=False)
(patch_embed): PatchEmbed(
  (proj): Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
)
(pos_drop): Dropout(p=0.0, inplace=False)
(time_drop): Dropout(p=0.0, inplace=False)
(blocks): ModuleList(  #************
  (0): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): Identity()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
  (1): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): DropPath()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
.
.
.
.
.
.
  (11): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): DropPath()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
)
(norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True) **** I want to remove this layer*****
(head): Linear(in_features=768, out_features=400, bias=True) **** I want to remove this layer*****

)
)
Specifically, I want to extract the outputs of the 4th, 8th and 11th blocks of the model and removing the lats two layers. How can I do this. I tried using TimeSformer.blocks[0] but that was not working.
Update :
I have a Class and I need to access the aforementioned blocks of the TimeSformer as the output of this class. The input of this class is a 5D tensor. This is the non-modified code that I use for extracting the outputs of the aforementioned blocks:
class Model(nn.Module):
def __init__(self, pretrained=False):
    super(Model, self).__init__()

    self.model =TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',
                                       pretrained_model='/home/user/models/TimeSformer_divST_16x16_448_K400.pyth')

    self.activation = {}
    def get_activation(name):
        def hook(model, input, output):
            self.activation[name] = output.detach()
            return hook

    self.model.model.blocks[4].register_forward_hook(get_activation('block4'))
    self.model.model.blocks[8].register_forward_hook(get_activation('block8'))
    self.model.model.blocks[11].register_forward_hook(get_activation('block11'))

    block4_output = self.activation['block4']
    block8_output = self.activation['block8']
    block11_output = self.activation['block11']

def forward(self, x, out_consp = False):

    features2, features3, features4 = self.model(x)

Output:
To extract the intermediate output from specific layers, you can register it as a hook, the example is showed by the snipcode below:
import torch
from timesformer.models.vit import TimeSformer

model = TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',  pretrained_model='/path/to/pretrained/model.pyth')

activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model.model.blocks[4].register_forward_hook(get_activation('block4'))
model.model.blocks[8].register_forward_hook(get_activation('block8'))
model.model.blocks[11].register_forward_hook(get_activation('block11'))

x = torch.randn(3,3,224,224)
output = model(x)

block4_output = activation['block4']
block8_output = activation['block8']
block11_output = activation['block11']

To remove the last two layers, you can replace them with Identity:
model.norm = torch.nn.Identity()
model.head= torch.nn.Identity()


Instruction: Supress UserWarning from torchmetrics

Input:
When I train a neural network using PyTorch, I get the following warning caused by the torchmetrics library:

/Users/dev/miniconda/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/prints.py:36:
UserWarning: Torchmetrics v0.9 introduced a new argument class
property called full_state_update that has not been set for this
class (SMAPE). The property determines if update by default needs
access to the full metric state. If this is not the case, significant
speedups can be achieved and we recommend setting this to False. We
provide an checking function from torchmetrics.utilities import check_forward_no_full_state that can be used to check if the
full_state_update=True (old and potential slower behaviour, default
for now) or if full_state_update=False can be used safely.

I tried to suppress this warning by using the warnings package in my script:
with warnings.catch_warnings():
    warnings.simplefilter("ignore")

However, the warning is still shown which is probably due to a function in prints.py of torchmetrics:
def _warn(*args: Any, **kwargs: Any) -> None:
    warnings.warn(*args, **kwargs)

Is it possible to get rid of this warning from my script without changing the library code?

Output:
Use -W argument to control how python deals with warnings, consider following simple example, let dowarn.py content be
import warnings
warnings.warn("I am UserWarning", UserWarning)
warnings.warn("I am FutureWarning", FutureWarning)

then
python dowarn.py

gives
dowarn.py:2: UserWarning: I am UserWarning
  warnings.warn("I am UserWarning", UserWarning)
dowarn.py:3: FutureWarning: I am FutureWarning
  warnings.warn("I am FutureWarning", FutureWarning)

and
python -W ignore dowarn.py

gives empty output and
python -W ignore::UserWarning dowarn.py

gives
dowarn.py:3: FutureWarning: I am FutureWarning
  warnings.warn("I am FutureWarning", FutureWarning)

See python man page for discussion of -W values


Instruction: Accessing a module inside of a block in a pretrained model

Input:
How can I access the output of a specific layer in a specific block of a pretrained model. To be clearer, the print of the TimeSformer model is as follows:
The print of the model is as follows:
TimeSformer(
  (model): VisionTransformer(
(dropout): Dropout(p=0.0, inplace=False)
(patch_embed): PatchEmbed(
  (proj): Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
)
(pos_drop): Dropout(p=0.0, inplace=False)
(time_drop): Dropout(p=0.0, inplace=False)
(blocks): ModuleList(
  (0): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): Identity()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
  (1): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(  # *********
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True) # @@@@@@@
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): DropPath()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
.
.
.
.
.
.
  (11): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): DropPath()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
)
(norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(head): Linear(in_features=768, out_features=400, bias=True)

)
Based on the answer that was proposed in this post it is possible to have the access to the output of a block:
    import torch
from timesformer.models.vit import TimeSformer

model = TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',  pretrained_model='/path/to/pretrained/model.pyth')

activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model.model.blocks[4].register_forward_hook(get_activation('block4'))
model.model.blocks[8].register_forward_hook(get_activation('block8'))
model.model.blocks[11].register_forward_hook(get_activation('block11'))

x = torch.randn(3,3,224,224)
output = model(x)

block4_output = activation['block4']
block8_output = activation['block8']
block11_output = activation['block11']

My question is that how can I have access to a module inside the blocks or a layer inside of a module of a block that are represented by * and @ signs. To be clearer, How can I have access the output of the (temporal_attn) and also output of the (proj) that is inside of the (temporal_attn).

Output:
Having access to those blocks, you can easily proceed with accessing the sub-modules via the dot notation assuming those blocks are custom nn.Module (i.e. they are not subscriptable and the bracket notation can't be used). For instance with block n°4:
>>> model.model.blocks[4].temporal_attn \
       .register_forward_hook(get_activation('attn_block4'))

>>> model.model.blocks[4].temporal_attn.proj \
       .register_forward_hook(get_activation('attn_proj_block4'))


Instruction: Convert a tensorflow script to pytorch (TransformedDistribution)

Input:
I am trying to rewrite a tensorflow script in pytorch. I have a problem finding the equivalent part in torch for the following line from this script:
import tensorflow_probability as tfp
tfd = tfp.distributions
a_distribution = tfd.TransformedDistribution(
        distribution=tfd.Normal(loc=0.0, scale=1.0),
        bijector=tfp.bijectors.Chain([
            tfp.bijectors.AffineScalar(shift=self._means,
                                       scale=self._mags),
            tfp.bijectors.Tanh(),
            tfp.bijectors.AffineScalar(shift=mean, scale=std),
        ]),
        event_shape=[mean.shape[-1]],
        batch_shape=[mean.shape[0]])

In particular, I have a huge problem for replacing the tfp.bijectors.Chain component.
I wrote the following lines in torch, but I am wondering whether these lines in pytorch compatible with the above tensorflow code and whether I can specify the batch_shape somewhere?
base_distribution = torch.normal(0.0, 1.0)
transforms = torch.distributions.transforms.ComposeTransform([torch.distributions.transforms.AffineTransform(loc=self._action_means, scale=self._action_mag, event_dim=mean.shape[-1]), torch.nn.Tanh(),torch.distributions.transforms.AffineTransform(loc=mean, scale=std, event_dim=mean.shape[-1])])
a_distribution = torch.distributions.transformed_distribution.TransformedDistribution(base_distribution, transforms)

Any solution?

Output:
In Pytorch, the base distribution class Distribution expects both a batch_shape and a  event_shape parameter. Now notice that the subclass TransformedDistribution does not take such parameters (src code). That's because they are inferred from the base distribution class provided on initialization: see here and here.
You already found out about AffineTransform and ComposeTransform. Keep in mind you must stick with classes from the torch.distributions.

This holds for torch.normal which should be replaced with Normal. With this class, the shape is inferred from the provided loc and scale tensors.

And nn.Tanh which should be replaced with TanhTransform.

Here is a minimal example using your transformation pipeline:
Imports:
from torch.distributions.normal import Normal
from torch.distributions import transforms as tT
from torch.distributions.transformed_distribution import TransformedDistribution

Parameters:
mean = torch.rand(2,2)
std = 1
_action_means, _action_mag = 0, 1
event_dim=mean.shape[-1]

Distribution definition:
a_distribution = TransformedDistribution(
    base_distribution=Normal(loc=torch.full_like(mean, 0),
                             scale=torch.full_like(mean, 1)),
    transforms=tT.ComposeTransform([
        tT.AffineTransform(loc=_action_means, scale=_action_mag, event_dim=event_dim),
        tT.TanhTransform(),
        tT.AffineTransform(loc=mean, scale=std, event_dim=event_dim)]))


Instruction: The essence of learnable positional embedding? Does embedding improve outcomes better?

Input:
I was recently reading the bert source code from the hugging face project. I noticed that the so-called "learnable position encoding" seems to refer to a specific nn.Parameter layer when it comes to implementation.
def __init__(self):
    super()
    positional_encoding = nn.Parameter()
def forward(self, x):
    x += positional_encoding

↑ Could be this feeling, then performed the learnable position encoding. Whether that means it's that simple or not, I'm not sure I understand it correctly, I want to ask someone with experience.
In addition, I noticed a classic bert structure whose location is actually coded only once at the initial input. Does this mean that the subsequent bert layers, for each other, lose the ability to capture location information?
BertModel(
  (embeddings): BertEmbeddings(
    (word_embeddings): Embedding(30522, 768, padding_idx=0)
    (position_embeddings): Embedding(512, 768)
    (token_type_embeddings): Embedding(2, 768)
    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  (encoder): BertEncoder(
    (layer): ModuleList(
      (0): BertLayer(...)
      ...
  (pooler): BertPooler(...)

Would I get better results if the results of the previous layer were re-positional encoded before the next BERT layer?

Output:
What is the purpose of positional embeddings?
In transformers (BERT included) the only interaction between the different tokens is done via self-attention layers. If you look closely at the mathematical operation implemented by these layers you will notice that these layers are permutation equivariant: That is, the representation of
"I do like coding"
and
"Do I like coding"
is the same, because the words (=tokens) are the same in both sentences, only their order is different.
As you can see, this "permutation equivariance" is not a desired property in many cases.
To break this symmetry/equivariance one can simply "code" the actual position of each word/token in the sentence. For example:
"I_1 do_2 like_3 coding_4"
is no longer identical to
"Do_1 I_2 like_3 coding_4"
This is the purpose of positional encoding/embeddings -- to make self-attention layers sensitive to the order of the tokens.
Now to your questions:

learnable position encoding is indeed implemented with a simple single nn.Parameter. The position encoding is just a "code" added to each token marking its position in the sequence. Therefore, all it requires is a tensor of the same size as the input sequence with different values per position.
Is it enough to introduce position encoding once in a transformer architecture?  Yes! Since transformers stack multiple self-attention layers it is enough to add positional embeddings once at the beginning of the processing. The position information is "fused" into the semantic representation learned per token.
A nice visualization of this effect in Vision Transformers (ViT) can be found in this work:
Shir Amir, Yossi Gandelsman, Shai Bagon and Tali Dekel Deep ViT Features as Dense Visual Descriptors (arXiv 2021).
In sec. 3.1 and fig. 3 they show how the position information dominates the representation of tokens at early layers, but as you go deeper in a transformer, semantic information takes over.


Instruction: Order-independent Deep Learning Model

Input:
I have a dataset with parallel time series. The column 'A' depends on columns 'B' and 'C'. The order (and the number) of dependent columns can change. For example:
            A   B    C
2022-07-23  1  10  100
2022-07-24  2  20  200
2022-07-25  3  30  300

How should I transform this data, or how should I build the model so the order of columns 'B' and 'C' ('A', 'B', 'C' vs 'A', C', 'B'`) doesn't change the result? I know about GCN, but I don't know how to implement it. Maybe there are other ways to achieve it.
UPDATE:
I want to generalize my question and make one more example. Let's say we have a matrix as a singe observation (no time series data):
   col1 col2  target
0     1    a      20
1     2    a      30
2     3    b      30
3     4    b      40

I would like to predict one value 'target' per each row/instance. Each instance depends on other instances. The order of rows is irrelevant, and the number of rows in each observation can change.

Output:
You are looking for a permutation invariant operation on the columns.
One way of achieving this would be to apply column-wise operation, followed by a global pooling operation.
How that achieves your goal:

column-wise operations are permutation equivariant; that is, applying the operation on the columns and permuting the output, is the same as permuting the columns and then applying the operation.
A global pooling operation (e.g., max-pool, avg-pool) across the columns is permutation invariant: the result of an average pool does not depend on the order of the columns.
Applying a permutation invariant operation on top of a permutation equivariant one results in an overall permutation invariant function.

Additionally, you should look at self-attention layers, which are also permutation equivariant.
What I would try is:

Learn a representation (RNN/Transformer) for a single time series. Apply this representation to A, B and C.
Learn a transformer between the representation of A to those of B and C: that is, use the representation of A as "query" and those of B and C as "keys" and "values".

This will give you a representation of A that is permutation invariant in B and C.

Update (Aug 3rd, 2022):
For the case of "observations" with varying number of rows, and fixed number of columns:
I think you can treat each row as a "token" (with a fixed dimension = number of columns), and apply a Transformer encoder to predict the target for each "token", from the encoded tokens.


Instruction: What is the difference between ... and : in Pytorch tensors and numpy indexing

Input:
I am trying to get used to Pytorch indexing. However I couldn't understand the difference between tensor[:,-1] (which should print the last column) and tensor[...,-1] which is printing different output (output2)
import torch
tensor = torch.rand([3,3,3,3])
print('Output1')
print(tensor[:,-1])
print('Output2')
print(tensor[...,-1])

Output:
It looks like the following indices are equivalent
tensor[:, -1] == tensor[:, -1, :, :]
tensor[..., -1] == tensor[:, :, :, -1]


Instruction: Printing the size of the input and output of all the layers of a pretrained model

Input:
I want to print the sizes of the inputs and outputs of all the layers of a pretrained model. I uae this pretrained model as self.feature in my class.
The print of this pretrained model is as follows:
TimeSformer(
  (model): VisionTransformer(
(dropout): Dropout(p=0.0, inplace=False)
(patch_embed): PatchEmbed(
  (proj): Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
)
(pos_drop): Dropout(p=0.0, inplace=False)
(time_drop): Dropout(p=0.0, inplace=False)
(blocks): ModuleList(
  (0): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): Identity()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
  (1): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(  # *********
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True) # @@@@@@@
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): DropPath()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
.
.
.
.
.
.
  (11): Block(
    (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (temporal_attn): Attention(
      (qkv): Linear(in_features=768, out_features=2304, bias=True)
      (proj): Linear(in_features=768, out_features=768, bias=True)
      (proj_drop): Dropout(p=0.0, inplace=False)
      (attn_drop): Dropout(p=0.0, inplace=False)
    )
    (temporal_fc): Linear(in_features=768, out_features=768, bias=True)
    (drop_path): DropPath()
    (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (mlp): Mlp(
      (fc1): Linear(in_features=768, out_features=3072, bias=True)
      (act): GELU()
      (fc2): Linear(in_features=3072, out_features=768, bias=True)
      (drop): Dropout(p=0.0, inplace=False)
    )
  )
)
(norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(head): Linear(in_features=768, out_features=400, bias=True)

)
This the the code of my class and my method for printing the size of layers:
class Class(nn.Module):
    def __init__(self, pretrained=False):
        super(Class, self).__init__()

        self.feature =TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',
                                           pretrained_model='path/to/the/weight.pyth')

def forward(self, x):
    for layer in self.feature:
        x = layer(x)
        print(x.size())
    return x

I'm using the following approach for printing
But I am facing this error:
TypeError: 'TimeSformer' object is not iterable

How can I print the sizes of all the layers?
Update:
using the Following code receives the error mentioned in the comment:
def forward(self, x, out_consp = False):
layers=list(self.featureExtractor.children())
for layer in layers:
    x = layer(x)
    print(x.size())
return x

Output:
You can use hooks to print the shape of the input and the output of each layer. You can use this code to do what you want.
def hook_function(module, input, output):
    print(f'{module.name} :')
    print(module)
    #print(module)
    if isinstance(input[0], tuple):
        print('input shapes:')
        for elem in input[0]:
            print(elem.shape)
    else:
        print(f'input shape: {input[0].shape}')
    if isinstance(output, tuple):
        print('output shapes:')
        for elem in output:
            print(elem.shape)
    else:
        print(f'output shape: {output.shape}')
    print('')

def set_names(net):
    def recurs(net,parent_name=None):
        for name, mod in net.named_children():
            if parent_name is not None:
                name = '_'.join([parent_name, name])
            recurs(mod, name)
            setattr(mod,'name',name)

    recurs(net)
def print_shapes(network, dummy_input_shape, device='cuda', eval=True):
    network = network.to(device)
    if eval:
        network.eval()
    else:
        network.train()
        assert dummy_input_shape[0] > 1
    #print(network)
    dummy = torch.randn(dummy_input_shape, device=device)
    set_names(network)
    handles = []
    def attach_hooks(net):
        leaf_layers = 0
        for mod in net.children():
            leaf_layers += 1
            attach_hooks(mod)
        if leaf_layers == 0:
            handles.append(net.register_forward_hook(hook_function))
    attach_hooks(network)
    network(dummy)
    # if needed
    for handle in handles:
        handle.remove()

Example:
network = TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',
                                           pretrained_model='path/to/the/weight.pyth')
# The behaviour of a forward function could be different during training
print_shapes(network,(1,3,224,224),'cpu', eval=True)
print_shapes(network,(2,3,224,224),'cpu', eval=False)

A snippet of the output includes a layer that is defined before 'temporal_norm1' layer in 'Block' module but called or executed later (norm1).
model_blocks_11_temporal_fc :
Linear(in_features=768, out_features=768, bias=True)
input shape: torch.Size([2, 1568, 768])
output shape: torch.Size([2, 1568, 768])

model_blocks_11_norm1 :
LayerNorm((768,), eps=1e-06, elementwise_affine=True)
input shape: torch.Size([16, 197, 768])
output shape: torch.Size([16, 197, 768])


Instruction: Reshaping the dimension of a tensor in PyTorch

Input: There is a tensor with the shape of [b, nt*nh*nw, dim]. The values of nt, nh, and nw are in hand. How can I reshape this tensor to the form of [b, dim, nt, nh, nw]? For example, how it is possible to reshape [2, 3x2x4, 512] to [2,512,3,2,4]?

Output:
It all depends on your data layout in memory.
However, assuming nt, nh, and nw are in the correct ordering in your underlying data tensor then you can do so by permuting and reshaping your tensor.
First swap dimensions to place dim as the 2nd axis using torch.transpose or torch.permute. Then reshape the tensor to the desired shape with torch.view or torch.reshape:
>>> x.transpose(1,2).view(b, dim, nt, nh, nw)


Instruction: Pytorch dataloaders : Bad file descriptor and EOF for workers>0

Input:
Description of the problem
I am encountering a strange behavior during a neural network training with Pytorch dataloaders made from a custom dataset. The dataloaders are set with workers=4, pin_memory=False.
Most of the time, the training finished with no problems.
Sometimes, the training stopped at a random moment with the following errors:

OSError: [Errno 9] Bad file descriptor
EOFError

It looks like the error occurs during socket creation to access dataloader elements.
The error disappears when I set the number of workers to 0, but I need to accelerate my training with multiprocessing.
What could be the source of the error ? Thank you !
The versions of python and libraries
Python 3.9.12, Pyorch 1.11.0+cu102
EDIT: The error is occurring only on clusters
Output of error file
Traceback (most recent call last):
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 145, in _serve
Epoch 17:  52%|█████▏    | 253/486 [01:00<00:55,  4.18it/s, loss=1.73]

Traceback (most recent call last):
  File "/my_directory/bench/run_experiments.py", line 251, in <module>
    send(conn, destination_pid)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 50, in send
    reduction.send_handle(conn, new_fd, pid)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 183, in send_handle
    with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM) as s:
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/socket.py", line 545, in fromfd
    return socket(family, type, proto, nfd)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/socket.py", line 232, in __init__
    _socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 9] Bad file descriptor

    main(args)
  File "/my_directory/bench/run_experiments.py", line 183, in main
    run_experiments(args, save_path)
  File "/my_directory/bench/run_experiments.py", line 70, in run_experiments
    ) = run_algorithm(algorithm_params[j], mp[j], ss, dataset)
  File "/my_directorybench/algorithms.py", line 38, in run_algorithm
    data = es(mp,search_space,  dataset, **ps)
  File "/my_directorybench/algorithms.py", line 151, in es
   data = ss.generate_random_dataset(mp,
  File "/my_directorybench/architectures.py", line 241, in generate_random_dataset
    arch_dict = self.query_arch(
  File "/my_directory/bench/architectures.py", line 71, in query_arch
    train_losses, val_losses, model = meta_net.get_val_loss(
  File "/my_directory/bench/meta_neural_net.py", line 50, in get_val_loss
    return self.training(
  File "/my_directorybench/meta_neural_net.py", line 155, in training
    train_loss = self.train_step(model, device, train_loader, epoch)
  File "/my_directory/bench/meta_neural_net.py", line 179, in train_step
    for batch_idx, mini_batch in enumerate(pbar):
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1207, in _next_data
    idx, data = self._get_data()
  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1173, in _get_data
    success, data = self._try_get_data()
  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1011, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/my_directory/.local/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 295, in rebuild_storage_fd
    fd = df.detach()
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 189, in recv_handle
    return recvfds(s, 1)[0]
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 159, in recvfds
    raise EOFError
EOFError

EDIT : The way data is accessed
    from PIL import Image
    from torch.utils.data import DataLoader

    # extract of code of dataset

        class Dataset():
           def __init__(self,image_files,mask_files):
              self.image_files = image_files
              self.mask_files = mask_files

           def __getitem__(self, idx):
              img = Image.open(self.image_files[idx]).convert('RGB')
              mask=Image.open(self.mask_files[idx]).convert('L')
              return img, mask

    # extract of code of trainloader

        train_loader = DataLoader(
                        dataset=train_dataset,
                        batch_size=4,
                        num_workers=4,
                        pin_memory=False,
                        shuffle=True,
                        drop_last=True,
                        persistent_workers=False,
                    )

Output:
I have finally found a solution. Adding this configuration to the dataset script works:
import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')

By default, the sharing strategy is set to 'file_descriptor'.
I have tried some solutions explained in :

this issue (increase shared memory, increase max number of opened file descriptors, torch.cuda.empty_cache() at the end of each epoch, ...)
and this other issue, that turns out to solve the problem

As suggested by @AlexMeredith, the error may be linked to the distributed filesystem (Lustre) that some clusters use. The error may also come from distributed shared memory.