Instruction: How to parallelize a training loop ever samples of a batch when CPU is only available in pytorch?

Input:
I want to parallelize over single examples or batch of example (in my situation is that I only have cpus, I have up to 112). I tried it but I get a bug that the losses cannot have the gradient out of separate processes (which entirely ruins my attempt). I still want to do it and it essential that after the multiproessing happens that I can do an optimizer step. How do I get around it? I made a totally self contained example:

import torch
import torch.nn as nn
from torch.optim.lr_scheduler import StepLR
from torch.utils.data import Dataset, DataLoader

from torch.multiprocessing import Pool

class SimpleDataSet(Dataset):

    def __init__(self, Din, num_examples=23):
        self.x_dataset = [torch.randn(Din) for _ in range(num_examples)]
        # target function is x*x
        self.y_dataset = [x**2 for x in self.x_dataset]

    def __len__(self):
        return len(self.x_dataset)

    def __getitem__(self, idx):
        return self.x_dataset[idx], self.y_dataset[idx]

def get_loss(args):
    x, y, model = args
    y_pred = model(x)
    criterion = nn.MSELoss()
    loss = criterion(y_pred, y)
    return loss

def get_dataloader(D, num_workers, batch_size):
    ds = SimpleDataSet(D)
    dl = DataLoader(ds, batch_size=batch_size, num_workers=num_workers)
    return dl

def train_fake_data():
    num_workers = 2
    Din, Dout = 3, 1
    model = nn.Linear(Din, Dout).share_memory()

    optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

    batch_size = 2
    num_epochs = 10
    # num_batches = 5
    num_procs = 5
    dataloader = get_dataloader(Din, num_workers, batch_size)
    scheduler = StepLR(optimizer, step_size=1, gamma=0.7)
    for epoch in range(num_epochs):
        for _, batch in enumerate(dataloader):
            batch = [(torch.randn(Din), torch.randn(Dout), model) for _ in batch]
            with Pool(num_procs) as pool:
                optimizer.zero_grad()

                losses = pool.map(get_loss, batch)
                loss = torch.mean(losses)
                loss.backward()

                optimizer.step()
            # scheduler
            scheduler.step()

if __name__ == '__main__':
    # start = time.time()
    # train()
    train_fake_data()
    # print(f'execution time: {time.time() - start}')

Error:
Traceback (most recent call last):
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3427, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-ea57e03ba088>", line 1, in <module>
    runfile('/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py', wdir='/Users/brando/ML4Coq/playground/multiprocessing_playground')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py", line 95, in <module>
    train_fake_data()
  File "/Users/brando/ML4Coq/playground/multiprocessing_playground/multiprocessing_cpu_pytorch.py", line 83, in train_fake_data
    losses = pool.map(get_loss, batch)
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/multiprocessing/pool.py", line 290, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/brando/anaconda3/envs/coq_gym/lib/python3.7/multiprocessing/pool.py", line 683, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[tensor(0.5237, grad_fn=<MseLossBackward>)]'. Reason: 'RuntimeError('Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).')'

I am sure I want to do this. How should I be doing this?

New attempt using DDP
"""
Based on: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

Note: as opposed to the multiprocessing (torch.multiprocessing) package, processes can use
different communication backends and are not restricted to being executed on the same machine.
"""
import torch
from torch import nn, optim
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP

import os

num_epochs = 5
batch_size = 8
Din, Dout = 10, 5
data_x = torch.randn(batch_size, Din)
data_y = torch.randn(batch_size, Dout)
data = [(i*data_x, i*data_y) for i in range(num_epochs)]

class OneDeviceModel(nn.Module):
    """
    Toy example for a model ran in parallel but not distributed accross gpus
    (only processes with their own gpu or hardware)
    """
    def __init__(self):
        super().__init__()
        self.net1 = nn.Linear(Din, Din)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(Din, Dout)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
    """
    # set up the master's ip address so this child process can coordinate
    # os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
    if torch.cuda.is_available():
        backend = 'nccl'
    # Initializes the default distributed process group, and this will also initialize the distributed package.
    dist.init_process_group(backend, rank=rank, world_size=world_size)

def cleanup():
    """ Destroy a given process group, and deinitialize the distributed package """
    dist.destroy_process_group()

def run_parallel_training_loop(rank, world_size):
    """
    Distributed function to be implemented later.

    This is the function that is actually ran in each distributed process.

    Note: as DDP broadcasts model states from rank 0 process to all other processes in the DDP constructor,
    you don’t need to worry about different DDP processes start from different model parameter initial values.
    """
    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    setup_process(rank, world_size)

    # create model and move it to GPU with id rank
    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()
    # ddp_model = DDP(model, device_ids=[rank])
    ddp_model = DDP(model)

    for batch_idx, batch in enumerate(data):
        x, y = batch
        loss_fn = nn.MSELoss()
        optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

        optimizer.zero_grad()
        outputs = ddp_model(x)
        labels = y.to(rank) if torch.cuda.is_available() else y
        # Gradient synchronization communications take place during the backward pass and overlap with the backward computation.
        loss_fn(outputs, labels).backward()  # When the backward() returns, param.grad already contains the synchronized gradient tensor.
        optimizer.step()  # TODO how does the optimizer know to do the gradient step only once?

    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # Destroy a given process group, and deinitialize the distributed package
    cleanup()

def main():
    print()
    print('running main()')
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # args
    world_size = mp.cpu_count()
    mp.spawn(run_parallel_training_loop, args=(world_size,), nprocs=world_size)

if __name__ == "__main__":
    print('starting __main__')
    main()
    print('Done!\a\n')

it seems it works but my question is in line 74 do I need to do this
    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()

or
    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel()

for it to work properly in multiple CPUs?

Serial is faster than parallel even if I have 112 cpu cores?
My current issue is that the cpu parallel job is slower than the serially running one when only cpus are available.

I want to know how to set up python and parallel cpus. e.g. if I have X cpus how many processes should I be running...X? or what? How do I choose this number, even if its heursitics rough.

related links from research:

https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836
How to use multiprocessing in PyTorch?
https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698/7
https://www.reddit.com/r/pytorch/comments/sm073v/how_to_parallelize_a_training_loop_ever_samples/

Output:
Torch will use multiple CPU to parallelize operations, so your serial is maybe using multi-core vectorization.
Take this simple example
import torch
c = 0;
for i in range(10000):
    A = torch.randn(1000, 1000, device='cpu');
    B = torch.randn(1000, 1000, device='cpu');
    c += torch.sum(A @ B)

No code is used to parallelize, however 80% of 12 CPUs with the default configuration.

You can use torch.set_num_threads to set intraop parallelism on CPU. In particular if you are running multiple process and you want each process to use a single CPU you may want to set in each process the intraop parallelism to 1.
However, parallelizing the operations has a cost, I am unable go into the implementation details but we can run a quick experiment that shows the overhead of using multiple threads.
import matplotlib.pyplot as plt
import numpy as np
import torch;
import time;
A = torch.randn(1000, 1000, device='cpu');
B = torch.randn(1000, 1000, device='cpu');
funcs = {
    'sin': lambda a,b: torch.sin(A),
    'tanh': lambda a,b: torch.tanh(A),
    'log': lambda a,b: torch.log(A),
    'matmul': lambda a,b: A @ B.T
}
t = np.zeros(20)
for k,f in funcs.items():
    for i in range(1, len(t) + 1):
        torch.set_num_threads(i)
        c = 0;
        t0 = time.time();
        for _ in range(100):
            f(A,B)
        tf = time.time()
        t[i-1] = (tf - t0)*i;
    plt.plot(np.arange(1, len(t)+1), t, '-o', label=k)
plt.xlabel('Number of threads')
plt.legend()
plt.ylabel('Core x time')

The operations tends to run faster with parallelism

But if we take the total CPU time, by multiplying by the number of threads, we see that the single thread version is more efficient.

If you are able to parallelize your experiment at a higher level, by running independent processes, you should try that with a single core for each process, otherwise each process will try to use all the CPUs and all of them will run very slowly because your system is overloaded.
Tweaking DDP example
I modified hyperparameters of your example scripts intentionally in a way that weights in favor of multi process.

comparably less initialization overhead
comparably less communication between processes

"""
Based on: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

Note: as opposed to the multiprocessing (torch.multiprocessing) package, processes can use
different communication backends and are not restricted to being executed on the same machine.
"""
import torch
from torch import nn, optim
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
import argparse
import os

# More than one epoch so that the initialization is less significant
# than compared to the model processing time
num_epochs = 10
# for the experiment select a number that has a lot of divisors
# as I want to test with equal number of batches
num_batches = 16*9*5
# Uses a larger batch so that more work is done in each process
# between two gradient synchronizations
# apparently the intraop optimization is not helping
# (at least not too much) in the batch dimension
batch_size = 10000
# Use smaller dimensions, so that the intraop parallelization becomes less
# helpful
Din, Dout = 3, 5
data_x = torch.randn(batch_size, Din)
data_y = torch.randn(batch_size, Dout)
data = [(i*data_x, i*data_y) for i in range(num_batches)]

class OneDeviceModel(nn.Module):
    """
    Toy example for a model ran in parallel but not distributed accross gpus
    (only processes with their own gpu or hardware)
    """
    def __init__(self):
        super().__init__()
        # -- Use more layers
        self.net = [nn.Linear(Din, Din) for _ in range(10)]
        # -- Bob: use more complex activation
        self.tanh = nn.Tanh()
        self.sigmoid = nn.Sigmoid()
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(Din, Dout)

    def forward(self, x):
      # apply the 10 layers sequentially
      for i in range(10):
        x = self.net[i](x)
        x = self.sigmoid(x)
        x = self.tanh(x)
        x = self.relu(x)
      return self.net2(x)

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
    """
    # set up the master's ip address so this child process can coordinate
    # os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
    if torch.cuda.is_available():
        backend = 'nccl'
    # Initializes the default distributed process group, and this will also initialize the distributed package.
    dist.init_process_group(backend, rank=rank, world_size=world_size)

def cleanup():
    """ Destroy a given process group, and deinitialize the distributed package """
    dist.destroy_process_group()

def run_parallel_training_loop(rank, world_size):
    """
    Distributed function to be implemented later.

    This is the function that is actually ran in each distributed process.

    Note: as DDP broadcasts model states from rank 0 process to all other processes in the DDP constructor,
    you don’t need to worry about different DDP processes start from different model parameter initial values.
    """
    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    setup_process(rank, world_size)
    torch.set_num_threads(mp.cpu_count() // world_size)
    # create model and move it to GPU with id rank
    model = OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()
    # ddp_model = DDP(model, device_ids=[rank])
    ddp_model = DDP(model)
    for _ in range(num_epochs):
      for batch_idx, batch in enumerate(data[rank::world_size]):
          x, y = batch
          loss_fn = nn.MSELoss()
          optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

          optimizer.zero_grad()
          outputs = ddp_model(x)
          labels = y.to(rank) if torch.cuda.is_available() else y
          # Gradient synchronization communications take place during the backward pass and overlap with the backward computation.
          loss_fn(outputs, labels).backward()  # When the backward() returns, param.grad already contains the synchronized gradient tensor.
          optimizer.step()  # TODO how does the optimizer know to do the gradient step only once?

    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # Destroy a given process group, and deinitialize the distributed package
    cleanup()

def main():
    print()
    print('running main()')
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    parser = argparse.ArgumentParser()
    parser.add_argument('--world-size', default=1, type=int)
    args = parser.parse_args()
    assert num_batches % args.world_size == 0
    mp.spawn(run_parallel_training_loop, args=(args.world_size,), nprocs=args.world_size)

if __name__ == "__main__":
    print('starting __main__')
    main()
    print('Done!\a\n')

$ time python3 ddp.py --world-size 1 > /dev/null

real    0m59.092s
user    8m46.589s
sys     0m7.320s

$ time python3 ddp.py --world-size 1 > /dev/null

real    1m11.124s
user    10m54.209s
sys     0m9.595s

$ time python3 ddp.py --world-size 6 > /dev/null

real    0m18.348s
user    2m28.799s
sys     0m18.068s
$ time python3 ddp.py --world-size 12 > /dev/null

real    0m26.352s
user    4m3.074s
sys     0m39.179s
$ time python3 ddp.py --world-size 3 > /dev/null

real    0m23.047s
user    3m51.172s
sys     0m11.483s
$ time python3 ddp.py --world-size 4 > /dev/null

real    0m18.195s
user    2m55.241s
sys     0m12.841s
$ time python3 ddp.py --world-size 2 > /dev/null

real    0m26.955s
user    4m15.837s
sys     0m7.127s

If I remove the line
torch.set_num_threads(mp.cpu_count() // world_size)

$ time python3 ddp.py --world-size 4 > /dev/null

real    0m40.574s
user    6m39.176s
sys     0m19.025s

$ time python3 ddp.py --world-size 2 > /dev/null

real    0m28.066s
user    3m17.775s
sys     0m8.410s

$ time python3 ddp.py --world-size 1 > /dev/null

real    0m37.114s
user    2m19.743s
sys     0m4.866s

Using
torch.set_num_threads(mp.cpu_count() // world_size // 2)

$ time python3 ddp.py --world-size 6 > /dev/null

real    0m16.399s
user    1m38.915s
sys     0m20.780s

$ time python3 ddp.py --world-size 4 > /dev/null

real    0m15.649s
user    1m1.821s
sys     0m13.589s

$ time python3 ddp.py --world-size 3 > /dev/null

real    0m16.947s
user    1m29.696s
sys     0m10.069s

$ time python3 ddp.py --world-size 2 > /dev/null

real    0m21.851s
user    2m4.564s
sys     0m7.486s

My Opinion
DDP in a single node seems not particularly advantageous. Unless you have a model that does a lot of work that is particularly not well handled by pytorch intraop parallelism, have large batches, and preferrably models with less parameters and more operations, meaning less gradients to synchronize, e.g. a convolutional model on a very large input.
Other scenarios where DDP might be helpful is if you are using too much python in your model, instead of vectorized operations.


Instruction: How to squeeze all but one torch dims?

Input:
torch.squeeze can convert the shape of a tensor to not have dimensions of size 1.
I want to squeeze my tensor in all dimensions but one (in this example, not squeeze dim=0).
All I can see in the doc is

dim (int, optional) – if given, the input will be squeezed only in
this dimension

I want the opposite:
t = torch.zeros(5, 1, 6, 1, 7, 1)

squeezed = torch.magic_squeeze(keep_dim=3)

assert squeezed == (5, 6, 1, 7)

Can this be done?

Output:
Reshape will let you accomplish what you want to do:
import torch

t = torch.zeros(5, 1, 6, 1, 7, 1)
t = t.reshape((5, 6, 1, 7))
>>> torch.Size([5, 6, 1, 7])


Instruction: PyTorch, select batches according to label in data column

Input:
I have a dataset like such:

index
tag
feature1
feature2
target

1
tag1
1.4342
88.4554
0.5365

2
tag1
2.5656
54.5466
0.1263

3
tag2
5.4561
845.556
0.8613

4
tag3
6.5546
8.52545
0.7864

5
tag3
8.4566
945.456
0.4646

The number of entries in each tag is not always the same.
And my objective is to load only the data with a specific tag or tags, so that I get only the entries in tag1 for one mini-batch and then tag2 for another mini-batch if I set batch_size=1. Or for instance tag1 and tag2 if I set batch_size=2
The code I have so far disregards completely the tag label and just chooses the batches randomly.
I built the datasets like such:
# features is a matrix with all the features columns through all rows
# target is a vector with the target column through all rows
featuresTrain, targetTrain = projutils.get_data(train=True, config=config)
train = torch.utils.data.TensorDataset(featuresTrain, targetTrain)
train_loader = make_loader(train, batch_size=config.batch_size)

And my loader (generically) looks like this:
def make_loader(dataset, batch_size):
loader = torch.utils.data.DataLoader(dataset=dataset,
                                     batch_size=batch_size,
                                     shuffle=True,
                                     pin_memory=True,
                                     num_workers=8)
return loader

Which I then train like this:
for epoch in range(config.epochs):
    for _, (features, target) in enumerate(loader):
        loss = train_batch(features, target, model, optimizer, criterion)

And the train_batch:
def train_batch(features, target, model, optimizer, criterion):
features, target = features.to(device), target.to(device)

# Forward pass ➡
outputs = model(features)
loss = criterion(outputs, target
return loss

Output:
A simple dataset that implements roughly the characteristics you're looking for as best as I can tell.
class CustomDataset(data.Dataset):
    def __init__(self,featuresTrain,targetsTrain,tagsTrain,sample_equally = False):
       # self.tags should be a tensor in k-hot encoding form so a 2D tensor,
       self.tags = tagsTrain
       self.x = featuresTrain
       self.y = targetsTrain
       self.unique_tagsets = None
       self.sample_equally = sample_equally

       # self.active tags is a 1D k-hot encoding vector
       self.active_tags = self.get_random_tag_set()

    def get_random_tag_set(self):
        # gets all unique sets of tags and returns one randomly
        if self.unique_tagsets is None:
             self.unique_tagsets = self.tags.unique(dim = 0)
        if self.sample_equally:
             rand_idx = torch.randint(len(self.unique_tagsets),[1])[1].detatch().int()
             return self.unique_tagsets[rand_idx]
        else:
            rand_idx = torch.randint(len(self.tags),[1])[1].detatch().int()
            return self.tags[rand_idx]

    def set_tags(self,tags):
       # specifies the set of tags that must be present for a datum to be selected
        self.active_tags = tags

    def __getitem__(self,index):
        # get all indices of elements with self.active_tags
        indices = torch.where(self.tags == self.active_tags)[0]

        # we select an index based on the indices of the elements that have the tag set
        idx = indices[index % len(indices)]

        item = self.x[idx], self.y[idx]
        return item

    def __len__(self):
        return len(self.y)

This dataset randomly selects a set of tags. Then, every time __getitem__() is called, it uses the index specified to select from amongst the data elements that have the set of tags. You can call set_tags() or get_random_tag_set() then set_tags() after each minibatch or however often you want to change up the tagset, or you can manually specify the tagset yourself. The dataset inherits from torch.data.Dataset so you should be able to use if with a torch.data.Dataloader without modification.
You can specify whether you'd like to sample each set of tags according to its prevalence, or whether you'd like to sample all tagsets equally regardless of how many elements have that set, using sample_equally.
In short, this dataset is a tiny bit rough around the edges but should allow you to sample batches all with the same tag set. The main shortcoming is that each element will likely be sampled more than once per batch.
For the initial encoding, let's say that to start each data example has a list of tags, so tags is a list of lists, each sublist containing tags. The following code would convert this to k-hot encoding, so you can just:
def to_k_hot(tags):
  all_tags = []
  for ex in tags:
    for tag in ex:
        all_tags.append(tag)
  unique_tags = list(set(all_tags)) # remove duplicates

  tagsTrain = torch.zeros([len(tags),len(unique_tags)]):
  for i in range(len(tags)): # index through all examples
    for j in range(len(unique_tags)): # index through all unique_tags
        if unique_tags[j] in tags[i]:
             tagsTrain[i,j] = 1

  return tagsTrain

As an example, say you had the following tags for a dataset:
tags = [ [tag1],
         [tag1,tag2],
         [tag3],
         [tag2],
         [],
         [tag1,tag2,tag3] ]

Calling to_k_hot(tags) would return:
tensor([1,0,0],
       [1,1,0],
       [0,0,1],
       [0,1,0],
       [0,0,0],
       [1,1,1]])


Instruction: Vanishing seq_len in attention-based BiLSTM

Input:
I'm studying several implementations of self attention-based BiLSTM and I don't understand why in each of them the input and output size are different. In particular I refer to the following codes taken from different implementations:
Implementation 1 e 2
 def attnetwork(self, encoder_out, final_hidden):
        # encoder_out shape = (batch_size, seq_len, n_hidden)
        # final_hidden shape = (1, batch_size, n_hidden)
        hidden = final_hidden.squeeze(0)
        attn_weights = torch.bmm(encoder_out, hidden.unsqueeze(2)).squeeze(2)
        soft_attn_weights = F.softmax(attn_weights, 1)
        new_hidden = torch.bmm(encoder_out.transpose(1,2), soft_attn_weights.unsqueeze(2)).squeeze(2)
        return new_hidden # shape = (batch_size, n_hidden)

As you can see this implementation takes as input two vectors of dimension (batch_size, seq_len, n_hidden) and (1, batch_size, n_hidden), respectively, and returns a vector of dimensions (batch_size, n_hidden) . But where is the dimension relative to seq_len? I need to have an output vector equal to the input one (i.e (batch_size, seq_len, n_hidden)).
Another implementation where the input size does not match the output size:
def attention(self,H):
        M = torch.tanh(H) # Non-linear transformation size:(batch_size, hidden_dim, seq_len)
        a = F.softmax(torch.bmm(self.att_weight,M),dim=2) # a.Size : (batch_size,1,seq_len)
        a = torch.transpose(a,1,2) # (batch_size,seq_len,1)
        return torch.bmm(H,a) # (batch_size,hidden_dim,1)

Another implementation with the same problem:
def attention(self, rnn_out, state):
        merged_state = torch.cat([s for s in state],1)
        merged_state = merged_state.squeeze(0).unsqueeze(2)
        # (batch, seq_len, cell_size) * (batch, cell_size, 1) = (batch, seq_len, 1)
        weights = torch.bmm(rnn_out, merged_state)
        weights = torch.nn.functional.softmax(weights.squeeze(2)).unsqueeze(2)
        # (batch, cell_size, seq_len) * (batch, seq_len, 1) = (batch, cell_size, 1)
        return torch.bmm(torch.transpose(rnn_out, 1, 2), weights).squeeze(2)

How could one do to output a tensor of the same size as the input one without "breaking" the self attention mechanism?
Thank you!
EDIT:
The forward function I have to use is this:
def forward(self, x, x_len):
        x = nn.utils.rnn.pack_padded_sequence(x, x_len, batch_first=True)
        out1, (h_n, c_n) = self.lstm1(x)
        # out1 = (seq_len, batch, num_directions * hidden_size)
        # h_n = (num_layers * num_directions, batch, hidden_size)
        x, lengths = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
        x, att1 = self.atten1(x, lengths)  # skip connect
        return x

the final x in return x I absolutely need it to have the shape (batch_size, seq_len, hidden_state) (obv also in another order so that a transpose is enough to fix it).

Output:
From my experience, what an attention-based model does is :

Calculate some relationships between decoder hidden states and
encoder outputs.
Take softmax to get the attention distribution.
Take a weighted sum of the encoder output to get the attention output.

And a Seq2seq model act like:

Pass your sequences e.g., an sentence through an encoder, get a encoder output and a hidden layer ,use this hidden layer to initialize your decoder.
Calculate your attention model by initial decoder hidden states and encoder outputs.
Feed your decoder with a start character e.g.,  ,get a single character as output and a new hidden state, then use your attention model to calculate a new output.
Feed your decoder with a single-character input (defined on your own, may depend on a teacher-force ratio) and attention output (may concatenate attention output with decoder hidden state or something else), then you get a new character.
Loop 4 to get your predict sequence character by character.

You may have a look at this (from page 59):
http://web.stanford.edu/class/cs224n/slides/cs224n-2021-lecture07-nmt.pdf
Every time a character sent to your LSTM decoder, it will output a new character, and its hidden states will update to another value. The key is, hidden states keep changing.
So a (batch_size,hidden_dim,1) attention output is reasonable, since you should give a character to your model, use a corresponding attention output to help predicting next character, then get a new hidden state and get a new attention output and so on. You may not give a whole sequence to your LSTM decoder, you can't use any hidden states under that circumstance.


Instruction: How perform unsupervised clustering on numbers in an Array using PyTorch

Input:
I got this array and I want to cluster/group the numbers into similar values.
An example of input array:
array([ 57, 58, 59, 60, 61, 78, 79, 80, 81, 82, 83, 101, 102, 103, 104, 105, 106]

expected result :
array([57,58,59,60,61]), ([78,79,80,81,82,83]), ([101,102,103,104,105,106])

I tried to use clustering but I don't think it's gonna work if I don't know how many I'm going to split up.
true = np.where(array>=1)
-> (array([ 57,  58,  59,  60,  61,  78,  79,  80,  81,  82,  83, 101, 102,
    103, 104, 105, 106], dtype=int64),)

Output:
You can perform kind of derivation on this array so that you can track changes better, assume your array is:
A = np.array([ 57, 58, 59, 60, 61, 78, 79, 80, 81, 82, 83, 101, 102, 103, 104, 105, 106])

so you can make a derivation vector by simply convolving your vector with [-1 1]:
A_ = abs(np.convolve(A, np.array([-1, 1])))

then A_ is:
array([57, 1, 1, 1, 1, 17, 1, 1, 1, 1, 1, 18, 2, 1, 1, 1, 106]

now you can define a threshold like 5 and find the cluster boundaries.
THRESHOLD = 5
cluster_bounds = np.argwhere(A_ > THRESHOLD)

now cluster_bounds is:
array([[0], [5], [11], [16]], dtype=int32)


Instruction: Pytorch tensor shape

Input:
I have a simple question regarding the shapes of 2 different tensors - tensor_1 and tensor_2.

tensor_1.shape outputs torch.Size([784, 1]);
tensor_2.shape outputs torch.Size([784]).

I understand that the first one is rank-2 tensor, whereas the second is rank-1. What's hard for me is to conceptualize the difference between shape [784, 1] and [784].
Is it correct to think that tensor_1 has 784 rows and 1 column with a scalar inside each place? If so, why can't we call it simply a column vector (which is, in fact, rank-1 tensor), which also has values displayed vertically?
Similarly, can the shape of the second tensor ([784]) be imagined as 784 values inside a horizontal vector?

Output:
You cant call tensor_1 as column vector because of its dimension . indexing that particular tensor is done in 2D
eg . tensor_1[1,1]
Coming to tensor_2 , its a scalar tensor having only one dimension.
And of course you can make it have a shape of tensor_1, just do
tensor_2 = tensor_2.unsqueeze(1)   #This method will make tensor_2 have a shape of tensor_1


Instruction: Testing my CNN on a small set of image but training has no effect

Input:
I constructed a CNN to recognize 9 classes of gestures in images of 224x224x3. I try to test its functionality by training it on 16 images and see if it overfits to 100 accuracy. Here is my network
    import torch.nn as nn
    class learn_gesture(nn.Module):
        def __init__(self):
            super(learn_gesture, self).__init__()
            self.name = "gesture_learner"
            self.conv1 = nn.Conv2d(in_channels=3, out_channels=20, kernel_size=5, stride=1, padding=2)
            self.conv2 = nn.Conv2d(in_channels=20, out_channels=50, kernel_size=5, stride=1, padding=2)
            self.conv3 = nn.Conv2d(in_channels=50, out_channels=100, kernel_size=5, stride=1, padding=2)
            self.conv4 = nn.Conv2d(in_channels=100, out_channels=200, kernel_size=5, stride=1, padding=2)
            self.conv5 = nn.Conv2d(in_channels=200, out_channels=400, kernel_size=5, stride=1, padding=2)
            self.pool1 = nn.MaxPool2d(2,2)
            self.pool2 = nn.MaxPool2d(2,2)
            self.pool3 = nn.MaxPool2d(2,2)
            self.pool4 = nn.MaxPool2d(2,2)
            self.pool5 = nn.MaxPool2d(2,2)
            self.fc1 = nn.Linear(7*7*400, 10000)
            self.fc2 = nn.Linear(10000, 3000)
            self.fc3 = nn.Linear(3000, 9)

        def forward(self, x):
            x = self.pool1(F.relu(self.conv1(x))) # gives 112*20
            x = self.pool2(F.relu(self.conv2(x))) # gives 56*50
            x = self.pool3(F.relu(self.conv3(x))) # gives 28*100
            x = self.pool4(F.relu(self.conv4(x))) # gives 14*200
            x = self.pool5(F.relu(self.conv5(x))) # gives 7*400
            x = x.view(-1, 7*7*400)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            return F.softmax(self.fc3(x), dim=1)

And here is the training code:
    overfit_model = learn_gesture()
    num_epochs = 200   #set it high so that it will converge
    ## loss function and optimizer
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(over_model.parameters(), lr=0.001, momentum=0.9)       #optimizer is SGD with momentum

    ## set up some empty np arrays to store our result for plotting later
    train_err = np.zeros(num_epochs)
    train_loss = np.zeros(num_epochs)
    ################################################ train the network
    for epoch in range(num_epochs):
        total_train_loss = 0
        total_train_err = 0
        total_epoch = 0
        for i, data in enumerate(smallLoader, 0):
            inputs, labels = data
            outputs = over_model(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            corr = (determine_corr(outputs, labels))    # get a list of bool representing right or wrong predictions in the batch
            total_train_err += corr.count(False)
            total_train_loss += loss.item()
            total_epoch += len(labels)
        train_err[epoch] = float(total_train_err) / total_epoch
        train_loss[epoch] = float(total_train_loss) / (i+1)
        print(("Epoch {}: Train err: {}, Train loss: {}").format(
                   enter code hereepoch + 1,
                   train_err[epoch],
                   train_loss[epoch]))

The training has no effect, and both the accuracy and loss has no improvement either. I just absolutely can't figure out where the error is. Any help is greatly appreciated!
###############  Update ##############
I got rid of the softmax in the forward function. Surprisingly, the performance of the model hasn't changed much. And I notice that some elements in the output now are negative and the elements across all classes do not add to 1. Is this supposed to happen?
output:
tensor([[ 0.0165, -0.0041,  0.0043,  0.0017,  0.0238,  0.0329, -0.0265, -0.0224,
     -0.0187],
    [ 0.0163, -0.0044,  0.0036,  0.0028,  0.0248,  0.0334, -0.0268, -0.0218,
     -0.0194],
    [ 0.0161, -0.0046,  0.0041,  0.0019,  0.0240,  0.0333, -0.0266, -0.0223,
     -0.0192],
    [ 0.0190, -0.0044,  0.0035,  0.0015,  0.0244,  0.0322, -0.0267, -0.0223,
     -0.0187],
    [ 0.0174, -0.0048,  0.0033,  0.0021,  0.0251,  0.0328, -0.0257, -0.0225,
     -0.0190],
    [ 0.0175, -0.0041,  0.0033,  0.0031,  0.0241,  0.0329, -0.0264, -0.0222,
     -0.0192],
    [ 0.0168, -0.0042,  0.0033,  0.0022,  0.0251,  0.0335, -0.0269, -0.0225,
     -0.0195],
    [ 0.0163, -0.0047,  0.0037,  0.0030,  0.0243,  0.0336, -0.0265, -0.0227,
     -0.0192],
    [ 0.0165, -0.0043,  0.0038,  0.0026,  0.0242,  0.0337, -0.0264, -0.0222,
     -0.0191],
    [ 0.0163, -0.0051,  0.0038,  0.0016,  0.0236,  0.0338, -0.0258, -0.0223,
     -0.0195],
    [ 0.0173, -0.0037,  0.0038,  0.0018,  0.0236,  0.0322, -0.0269, -0.0225,
     -0.0191],
    [ 0.0174, -0.0044,  0.0031,  0.0019,  0.0241,  0.0334, -0.0266, -0.0224,
     -0.0200],
    [ 0.0164, -0.0038,  0.0034,  0.0029,  0.0245,  0.0342, -0.0269, -0.0225,
     -0.0200],
    [ 0.0173, -0.0046,  0.0036,  0.0021,  0.0245,  0.0328, -0.0264, -0.0221,
     -0.0192],
    [ 0.0168, -0.0046,  0.0034,  0.0025,  0.0248,  0.0336, -0.0262, -0.0222,
     -0.0194],
    [ 0.0166, -0.0051,  0.0033,  0.0015,  0.0234,  0.0331, -0.0270, -0.0218,
     -0.0186]], grad_fn=<AddmmBackward>)
Epoch 199: Train err: 0.8125, Train loss: 2.1874701976776123

Output:
It seems that you are using a model named overfit_model where you pass over_model.parameters() to the optimizer:
optimizer = optim.SGD(over_model.parameters(), lr=0.001, momentum=0.9)

Should be replaced with ovrefit_model.parameters().

You are setting your gradients to zeros right after you back propagate, where it should be done beforehand. So, the following lines:
     loss.backward()
     optimizer.step()
     optimizer.zero_grad()

Should be replaced with:
     optimizer.zero_grad()
     loss.backward()
     optimizer.step()

There is no need to call F.softmax in
return F.softmax(self.fc3(x), dim=1)

since you are using nn.CrossEntropyLoss that calls F.cross_entropy which natively bundles log_softmax before calling nll_loss
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)


Instruction: Pytorch how use a linear activation function

Input:
In Keras, I can create any network layer with a linear activation function as follows (for example, a fully-connected layer is taken):
model.add(keras.layers.Dense(outs, input_shape=(160,), activation='linear'))

But I can't find the linear activation function in the PyTorch documentation. ReLU is not suitable, because there are negative values in my sample. How do I create a layer with a linear activation function in PyTorch?

Output:
If you take a look at the Keras documentation, you will see tf.keras.layers.Dense's activation='linear' corresponds to the a(x) = x function. Which means no non-linearity.
So in PyTorch, you just define the linear function without adding any activation layer:
torch.nn.Linear(160, outs)


Instruction: Append a tensor vector to tensor matrix

Input:
I have a tensor matrix that i simply want to append a tensor vector as another column to it.
For example:
    X = torch.randint(100, (100,5))
    x1 = torch.from_numpy(np.array(range(0, 100)))

I've tried torch.cat([x1, X) with various numbers for both axis and dim but it always says that the dimensions don't match.

Output:
You can also use torch.hstack to combine and unsqueeze for reshape x1
torch.hstack([X, x1.unsqueeze(1)])


Instruction: How to build a dataset from a large text file without getting a memory error?

Input:
I have a text file with size > 7.02 GB. I have already built a tokenizer based on this text file.  I want to build a dataset like so:
from transformers import LineByLineTextDataset

dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="data.txt", block_size=128,)

Since the size of my data is very large, a memory error occurs. This is the source code:
with open(file_path, encoding="utf-8") as f:
        lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]

    batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)
    print(batch_encoding)
    self.examples = batch_encoding["input_ids"]
    self.examples = [{"input_ids": torch.tensor(e, dtype=torch.long)} for e in self.examples]

Supposing that my text file has only 4 lines, the following will be printed:
{'input_ids': [[49, 93, 1136, 1685, 973, 363, 72, 3130, 16502, 18], [44, 73, 1685, 279, 7982, 18, 225], [56, 13005, 1685, 4511, 3450, 18], [56, 19030, 1685, 7544, 18]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1]]}

I have changed the source code as the following so that the memory error doesn't appear:
for line in open(file_path, encoding="utf-8"):
        if (len(line) > 0 and not line.isspace()):
            new_line = line.split()

            batch_encoding = tokenizer(new_line, add_special_tokens=True, truncation=True, max_length=block_size)
            print(batch_encoding)
            print(type(batch_encoding))
            self.examples = batch_encoding["input_ids"]
            self.examples = [{"input_ids": torch.tensor(e, dtype=torch.long)} for e in self.examples]
print(batch_encoding)

However, the following will be printed:
{'input_ids': [[49, 93], [3074], [329], [2451, 363, 72, 3130, 16502, 18]], 'token_type_ids': [[0, 0], [0], [0], [0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1], [1], [1], [1, 1, 1, 1, 1, 1]]}
<class 'transformers.tokenization_utils_base.BatchEncoding'>
{'input_ids': [[44, 73], [329], [69], [23788, 18]], 'token_type_ids': [[0, 0], [0], [0], [0, 0]], 'attention_mask': [[1, 1], [1], [1], [1, 1]]}
<class 'transformers.tokenization_utils_base.BatchEncoding'>
{'input_ids': [[56, 13005], [329], [7522], [7958, 18]], 'token_type_ids': [[0, 0], [0], [0], [0, 0]], 'attention_mask': [[1, 1], [1], [1], [1, 1]]}
<class 'transformers.tokenization_utils_base.BatchEncoding'>
{'input_ids': [[56, 19030], [329], [11639, 18]], 'token_type_ids': [[0, 0], [0], [0, 0]], 'attention_mask': [[1, 1], [1], [1, 1]]}
{'input_ids': [[56, 19030], [329], [11639, 18]], 'token_type_ids': [[0, 0], [0], [0, 0]], 'attention_mask': [[1, 1], [1], [1, 1]]}

How can I change the source code in order to be able to read the large text file line by line but get the same output as desired without a memory error?

Output:
You can create a dictionary storing the byte offsets for each line of the .txt file:
offset_dict = {}

with open(large_file_path, 'rb') as f:
    f.readline()  # move over header
    for line in range(number_of_lines):
        offset = f.tell()
            offset_dict[line] = offset

and then implement your own hashed __getitem__ method in a PyTorch Dataset (which can then be accessed by a DataLoader):
class ExampleDataset(Dataset):
    def __init__(self, large_file_path, offset_dict, ):
        self.large_file_path = large_file_path
        self.offset_dict = offset_dict

    def __len__(self):
        return len(self.offset_dict)

    def __getitem__(self, line):
        offset = self.offset_dict[line]
        with open(self.large_file_path, 'r', encoding='utf-8') as f:
            f.seek(offset)
            line = f.readline()
            return line


Instruction: TypeError: img should be PIL Image. Got

Input:
import torch
import torch.nn as nn

import torchvision.transforms.functional as TF

class DoubleConv(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(DoubleConv, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(in_channels, out_channels, 3, 1, 1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_channels, out_channels, 3, 1, 1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        return self.conv(x)

class UNET(nn.Module):
    def __init__(
            self, in_channels=3, out_channels=1, features=[64, 128, 256, 512],
    ):
        super(UNET, self).__init__()
        self.ups = nn.ModuleList()
        self.downs = nn.ModuleList()
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

        # Down part of UNET
        for feature in features:
            self.downs.append(DoubleConv(in_channels, feature))
            in_channels = feature

        # Up part of UNET
        for feature in reversed(features):
            self.ups.append(
                nn.ConvTranspose2d(
                    feature*2, feature, kernel_size=2, stride=2,
                )
            )
            self.ups.append(DoubleConv(feature*2, feature))

        self.bottleneck = DoubleConv(features[-1], features[-1]*2)
        self.final_conv = nn.Conv2d(features[0], out_channels, kernel_size=1)

    def forward(self, x):
        skip_connections = []

        for down in self.downs:
            x = down(x)
            skip_connections.append(x)
            x = self.pool(x)

        x = self.bottleneck(x)
        skip_connections = skip_connections[::-1]

        for idx in range(0, len(self.ups), 2):
            x = self.ups[idx](x)
            skip_connection = skip_connections[idx//2]

            if x.shape != skip_connection.shape:
                x = TF.resize(x, size=skip_connection.shape[2:])
            concat_skip = torch.cat((skip_connection, x), dim=1)
            x = self.ups[idx+1](concat_skip)

        return self.final_conv(x)

def test():
    x = torch.randn((3, 1, 161, 161))
    model = UNET(in_channels=1, out_channels=1)
    preds = model(x)
    #print(preds.shape)
    print(x.shape)
    assert preds.shape == x.shape

if __name__ == "__main__":
    test()

This is my code for UNet. The error massage I am getting is:
raise TypeError('img should be PIL Image. Got {}'.format(type(img)))

TypeError: img should be PIL Image. Got <class 'torch.Tensor'>

The error mentioning this line x = TF.resize(x, size=skip_connection.shape[2:])
What's wrong I am doing here?

Output:
As stated in the error you recievced, Tf.resize expects an input of type PIL.Image
Pil is a python library found here, it's used for dealing and processing images.
https://python-pillow.org/
Docs for the function in pytorch:
https://pytorch.org/vision/0.8/_modules/torchvision/transforms/functional.html#resize


Instruction: kmeans clustering python

Input:
There are coordinates that I want to cluster. The result of clustering using the kmeans
[[0, 107], [0, 108], [0, 109], [0, 115], [0, 116],
[0, 117], [0, 118], [0, 125], [0, 126], [0, 127],
[0, 128], [0, 135], [0, 136], [0, 194], [0, 195],
[1, 107], [1, 108], [1, 109], [1, 110], [1, 114],
[1, 115], [1, 116], [1, 117], [1, 118], [1, 119]...]

The result of clustering using the kmeans
from sklearn.cluster import KMeans
num_clusters = 9
km = KMeans(n_clusters=num_clusters)
km_fit = km.fit(nonzero_pred_sub)

>>>array([7, 7, 7, 1, 1, 1, 1, 5, 5, 5, 5, 3, 3, 0, 0, 7, 7, 7, 7, 1, 1, 1,
   1, 1, 1, 5, 5, 5...]

I want to know the coordinates of i-th cluster
for example, i need elements of 1st cluster and i can assume [0, 107], [0, 108], [0, 109] was clustered into the 7-th cluster.
How can i get coordinates from cluster?

Output:
I assume you want the coordinates affected to the 7th cluster. You can do so by storing you result in a dictionary :
from sklearn.cluster import KMeans
km = KMeans(n_clusters=9)
km_fit = km.fit(nonzero_pred_sub)

d = dict() # dictionary linking cluster id to coordinates
for i in range(len(km_fit)):
  cluster_id = km_fit[i]

  if cluster_id not in d:
    d[cluster_id] = []

  d[cluster_id].append(nonzero_pred_sub[i])

# that way you can access the 7th cluster coordinates like this
d[7]

>>> [[0, 107], [0, 108], [0, 109], [1, 107], [1, 108], [1, 109], [1, 110], ...]

To remove the "if" section in the loop, you can try looking into defaultdict objects.
You can also surely manage that with pandas dataframes to make manipulating more complex results easier.
If I misunderstood you question, and what you want is the coordinates of the center of the i-th cluster, you can get this by calling km_fit.cluster_centers_[i] (cf. doc).


Instruction: pytorch KAIR example on Android

Input:
I stuck trying to trace/scipt ffdnet KAIR's model to android. Model's forward looks like:
def forward(self, x): #, paddingBottom, paddingRight): #, sigma):

    noise_level_model = 15
    sigma = torch.full((1, 1, 1, 1), noise_level_model / 255.).type_as(x)

    h, w = x.size()[-2:]
    paddingBottom = int(np.ceil(h/2)*2-h)
    paddingRight = int(np.ceil(w/2)*2-w)

    x = torch.nn.ReplicationPad2d((0, paddingRight, 0, paddingBottom))(x)

    x = self.m_down(x)
    # m = torch.ones(sigma.size()[0], sigma.size()[1], x.size()[-2], x.size()[-1]).type_as(x).mul(sigma)
    m = sigma.repeat(1, 1, x.size()[-2], x.size()[-1])
    x = torch.cat((x, m), 1)
    x = self.model(x)
    x = self.m_up(x)

    x = x[..., :h, :w]
    return x

If I trace that I get some warnings about padding arguments but model works on Android. Problem is that it isn't work with input of different sizes, only size same as 'test1.jpeg':
model_name = 'ffdnet_color'
model_pool = 'model_zoo'
model_path = os.path.join(model_pool, model_name + '.pth')
n_channels = 3
nc = 96
nb = 12

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = net(in_nc=n_channels, out_nc=n_channels, nc=nc, nb=nb, act_mode='R')
model.load_state_dict(torch.load(model_path), strict=True)
model.eval()
for k, v in model.named_parameters():
    v.requires_grad = False
model = model.to(device)

img = 'testsets/myset/test1.jpeg'
img_name, ext = os.path.splitext(os.path.basename(img))
img_L = util.imread_uint(img, n_channels=n_channels)
img_L = util.uint2single(img_L)
noise_level_model = 15
img_L = util.single2tensor4(img_L)
img_L = img_L.to(device)
sigma_ = torch.full((1, 1, 1, 1), noise_level_model / 255)
sigma = torch.full((1, 1, 1, 1), noise_level_model / 255.).type_as(img_L)

traced_model = torch.jit.trace(model, img_L)
traced_optimized = optimize_for_mobile(traced_model)

save_path = os.path.splitext(os.path.basename(model_path))[0] + '-mobile.pth'
traced_optimized.save(save_path)

I've tried to script model with traced_model = torch.jit.script(model) but got errors:
TypeError: cannot create weak reference to 'numpy.ufunc' object

What should I do to achieve model works with different input size on mobile?

Output: I encountered a similar issue. It was due to my model using numpy math operations (which are numpy.ufunc). I fixed the issue by replacing all of numpy ufuncs (i.e. np.add, np.ceil, and +, - etc on ndarrays) with corresponding torch versions (i.e. torch.add, torch.sub etc).


Instruction: How to find r2score of my PyTorch model for regression

Input:
I have a UNet model. I'm trying for a regression model since, in my output, I have different floating values for each pixel. In order to check the r2score, I tried to put the below code in the model class, training_step, validation_step, and test_step.
from pytorch_lightning.metrics.functional import r2score
r2 = r2score(pred, y)
self.log('r2:',r2)
But it's giving the following error

ValueError: Expected both prediction and target to be 1D or 2D tensors, but recevied tensors with dimension torch.Size([50, 1, 32, 32])

How can I check my model fit?

Output:
The issue is that the function accepts 1D or 2D tensors, but your tensor is 4D (B x C x H x W). So to use the function you should reshape it:
r2 = r2score(pred.view(pred.shape[1], -1), y.view(y.shape[1], -1))


Instruction: How to create a tensor by accessing specific values at given indices of a 2 X 2 tensor in pytorch?

Input:
Suppose
mat = torch.rand((5,7)) and I want to get values from 1st dimension (here, 7) by passing the indices, say idxs=[0,4,2,3,6]. The way I am able to do it now is by doing mat[[0,1,2,3,4],idxs]. I expected mat[:,idxs] to work, but it didn't. Is the first option the only way or is there a better way?

Output:
torch.gather is what you are looking for:
torch.gather(mat, 1, torch.tensor(idxs)[:, None])


Instruction: Pytorch N - Beats model throwing error: 'str' object has no attribute '__name__'

Input:
I'm trying to replicate pytorch's N - Beats model in colab. I copied the same code from https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/ar.html to a colab notebook. There is an error showing up at training cell.
import os
import warnings
warnings.filterwarnings("ignore")
os.chdir("../../..")

import pandas as pd
import pytorch_lightning as pl
from pytorch_lightning.callbacks import EarlyStopping
import torch

from pytorch_forecasting import Baseline, NBeats, TimeSeriesDataSet
from pytorch_forecasting.data import NaNLabelEncoder
from pytorch_forecasting.data.examples import generate_ar_data
from pytorch_forecasting.metrics import SMAPE

data = generate_ar_data(seasonality=10.0, timesteps=400, n_series=100, seed
 = 42)
data["static"] = 2
data["date"] = pd.Timestamp("2020-01-01") + pd.to_timedelta(data.time_idx, "D")
data.head()

# create dataset and dataloaders
max_encoder_length = 60
max_prediction_length = 20

training_cutoff = data["time_idx"].max() - max_prediction_length

context_length = max_encoder_length
prediction_length = max_prediction_length

training = TimeSeriesDataSet(
data[lambda x: x.time_idx <= training_cutoff],
time_idx="time_idx",
target="value",
categorical_encoders={"series": NaNLabelEncoder().fit(data.series)},
group_ids=["series"],
# only unknown variable is "value" - and N-Beats can also not take any additional variables
 time_varying_unknown_reals=["value"],
 max_encoder_length=context_length,
 max_prediction_length=prediction_length,
)

validation = TimeSeriesDataSet.from_dataset(training, data,     min_prediction_idx=training_cutoff + 1)
batch_size = 128
train_dataloader = training.to_dataloader(train=True,     batch_size=batch_size, num_workers=0)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size, num_workers=0)

Error is:
---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-67-db4b0ef13391> in <module>()
     25     net,
     26     train_dataloader=train_dataloader,
---> 27     val_dataloaders=val_dataloader,
     28 )

30 frames

/usr/local/lib/python3.7/dist-packages/yaml/representer.py in     represent_object(self, data)
    329         if dictitems is not None:
    330             dictitems = dict(dictitems)
--> 331         if function.__name__ == '__newobj__':
    332             function = args[0]
    333             args = args[1:]

 AttributeError: 'str' object has no attribute '__name__'

Output: downgrading pytorch-lightning from 1.2.1 to 1.1.8 solved it for me.


Instruction: creating a progress bar for a validation test Python

Input:
The following code is used to find p-values for a significant test on Cifar-10 database.
Because we need a min of 1000 permutations, it is a very slow process, and I want to inlude a progress bar to show how time for each permutation.
I was thinking of using the tqdm library and sleep function, but am stuck on where to go from here. any help would be greatly appreciated!
'''
def validate_significance(val_loader, model, criterion, args):
model.eval()

vec_acc1 = []
vec_acc1_chance = []
vec_acc5 = []
vec_acc5_chance = []

for ss in range(0, args.num_permutations):

    val_loader = get_rand_sample_loader(val_loader.dataset, args)

    acc1_over_one_permutaion = 0
    acc1_chance_over_one_permutation = 0
    acc5_over_one_permutaion = 0
    acc5_chance_over_one_permutation = 0
    cnt = 0
    with torch.no_grad():
        for i, (images, target) in enumerate(val_loader):

            if args.gpu is not None:
                images = images.cuda(args.gpu, non_blocking=True)
            if torch.cuda.is_available():
                target = target.cuda(args.gpu, non_blocking=True)

            # compute output
            output = model(images)

            # measure accuracy on true labels
            acc1, acc5 = accuracy(output, target, topk=(1, 5))
            acc1_over_one_permutaion += acc1.item()
            acc5_over_one_permutaion += acc5.item()

            # now, measure accuracy on permuted labels
            target = target[torch.randperm(len(target))]  # scrambling the labels
            acc1_perm, acc5_perm = accuracy(output, target, topk=(1, 5))
            acc1_chance_over_one_permutation += acc1_perm.item()
            acc5_chance_over_one_permutation += acc5_perm.item()
            cnt += 1

    vec_acc1.append(acc1_over_one_permutaion / cnt)
    vec_acc1_chance.append(acc1_chance_over_one_permutation / cnt)
    vec_acc5.append(acc5_over_one_permutaion / cnt)
    vec_acc5_chance.append(acc5_chance_over_one_permutation / cnt)

p_acc1 = stats.ttest_ind(vec_acc1, vec_acc1_chance, equal_var=False)
p_acc5 = stats.ttest_ind(vec_acc5, vec_acc5_chance, equal_var=False)

return p_acc1, p_acc5

'''

Output:
from tqdm import tqdm

def validate_significance(val_loader, model, criterion, args):
    model.eval()

    vec_acc1 = []
    vec_acc1_chance = []
    vec_acc5 = []
    vec_acc5_chance = []

    for ss in tqdm(range(0, args.num_permutations)):
        ....


Instruction: Pytorch C++ API : CMake Issue

Input:
I want to include the pytorch C++ API to the large C++ software I am working on.
For legacy reasons, I must use find_package  and the associated find_path and find_library functions, instead of the proposed target_link_libraries.
Here's my FindTORCH.cmake :
include( FindPackageHandleStandardArgs )

find_path( TORCH_INCLUDE_DIR torch/torch.h
           PATHS
           /path/to/libtorch/include/torch/csrc/api/include/
           NO_DEFAULT_PATH )

find_library( TORCH_LIBRARIES libtorch.so
              PATHS
              /path/to/libtorch/lib/
              NO_DEFAULT_PATH )

FIND_PACKAGE_HANDLE_STANDARD_ARGS( TORCH REQUIRED_VARS TORCH_INCLUDE_DIR TORCH_LIBRARIES )

if ( TORCH_FOUND )
message( STATUS "Torch found" )
endif( TORCH_FOUND )

mark_as_advanced( TORCH_LIBRARIES TORCH_INCLUDE_DIR )

At the compilation, the torch files are found and I can include <torch/torch.h> in a random .cxx in the project.
However, if I add to the .cxx :
torch::Tensor tensor = torch::rand({2, 3});
cout << tensor << std::endl;

Then I cannot compile anymore and I get the following error :
/path/to/libtorch/include/torch/csrc/api/include/torch/utils.h:4:10: fatal error: ATen/record_function.h: No such file or directory
 #include <ATen/record_function.h>
          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

I'm working Ubuntu 18, C++ 14, and the cmake version is 3.10.2 .
Thanks in advance

Output:
Torch exposes its own targets. To use them effectively, simply remove FindTORCH.cmake from your project, and add /path/to/libtorch/ to your prefix path:
cmake_minimum_required(VERSION 3.19) # or whatever version you use
project(your-project CXX)

list(APPEND CMAKE_PREFIX_PATH "/path/to/libtorch/")
find_package(Torch REQUIRED CONFIG) # this ensure it find the file provided by pytorch

add_executable(your-executable main.cpp)

target_link_libraries(your-executable PUBLIC torch::Tensor)

If you really insist to use your own FindTorch.cmake instead of the correct one, you can modify it to create an imported target that you will then link:
You can change your find module very slightly to gain a modern CMake interface from it:
include( FindPackageHandleStandardArgs )

find_path( TORCH_INCLUDE_DIR torch/torch.h
           PATHS
           /path/to/libtorch/include/torch/csrc/api/include/
           NO_DEFAULT_PATH )

find_library( TORCH_LIBRARIES libtorch.so
              PATHS
              /path/to/libtorch/lib/
              NO_DEFAULT_PATH )

FIND_PACKAGE_HANDLE_STANDARD_ARGS( TORCH REQUIRED_VARS TORCH_INCLUDE_DIR TORCH_LIBRARIES )

if ( TORCH_FOUND )
    message( STATUS "Torch found" )
    add_library(torch::Tensor SHARED IMPORTED) # mimic the names from pytorch maintainers
    set_target_properties(torch::Tensor
    PROPERTIES
        IMPORTED_LOCATION "${TORCH_LIBRARIES}"
        INTERFACE_INCLUDE_DIRECTORIES "${TORCH_INCLUDE_DIR}"
        # on windows, set IMPORTED_IMPLIB to the .lib
    )
endif( TORCH_FOUND )

mark_as_advanced( TORCH_LIBRARIES TORCH_INCLUDE_DIR )

Then, in your main CMake file, you can use the imported target like any other targets:
find_package(Torch REQUIRED)

add_executable(your-executable main.cpp)
target_link_libraries(your-executable PUBLIC torch::Tensor)


Instruction: Why does Python's cProfile report a different elapsed time than using time.time() deltas when using PyTorch?

Input:
I am profiling some code using PyTorch. I am aware the CUDA normally has some asynchronous execution (see PyTorch docs), but I believe that transferring from GPU to CPU will generally force synchronization.
For this reason, I decided to naively profile using cProfile, but I notice that the time reported by Profile.enable() ... Profile.disable() was different than the time recorded across time.time() (as a delta).
Here's what the code looks like at a high-level:
gpu = torch.device("cuda")
cpu = torch.device("cpu")

setup = Setup()
net = make_fcn_resnet50(num_classes=setup.D)
net.eval().to(gpu)

rgb_tensor = setup.sample(device=cpu)

pr = profile.Profile()
pr.enable()
t_start = time.time()

rgb_tensor = rgb_tensor.to(gpu)
y = net(rgb_tensor)
dd_tensor = y["out"]
dd_mean = torch.mean(dd_tensor[[0]]).to(cpu).numpy()
assert dd_mean is not None

dt = time.time() - t_start
pr.disable()
stats = pstats.Stats(pr)

stats.print_stats(5)
print(f"dt: {dt:.4f}s")

Here's the discrepancy I see:
2925 function calls (2734 primitive calls) in 0.009 seconds
...
dt: 0.0355s

I would've expected cProfile to report about 35ms (same as dt), but instead it reports about 10ms.
Why does this happen?
Full code + repro is here:
https://github.com/EricCousineau-TRI/repro/tree/bdef8a14/python/cprofile_with_torch

Output:
Empirically, it seems like cProfile will not "hook" into the code if you don't "flush" all of the outputs, or if you're code isn't fully wrapped in a function.
See more detail in the comments here:
https://github.com/EricCousineau-TRI/repro/blob/bdef8a14b5/python/cprofile_with_torch/repro.py#L75-L94
All timing results recorded with:

Ubuntu 18.04
CPython 3.6.9
nvidia-driver-450 (450.102.04-0ubuntu0.18.04.1)
NVidia Titan RTX

That all being said, may be better just to use PyTorch's provided mechanisms (:facepalm:):

https://pytorch.org/tutorials/beginner/profiler.html
https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html


Instruction: CUDA initialization: Unexpected error from cudaGetDeviceCount()

Input:
I was running a deep learning program on my Linux server and I suddenly got this error.
UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729096996/work/c10/cuda/CUDAFunctions.cpp:100.)
Earlier when I just created this conda environment, torch.cuda.is_available() returned true and I could use CUDA & GPU. But all of a sudden I could not use CUDA and torch.cuda.is_available()returned false. What should I do?
ps. I use GeForce RTX 3080 and cuda 11.0 + pytorch 1.7.0. It worked before but now it doesn't.

Output: I just tried rebooting. Problem solved. Turned out that it was caused by NVIDIA NVML Driver/library version mismatch.


Instruction: from ._nnls import nnls ImportError: DLL load failed: The specified module could not be found

Input:
While running a UNet traning code I found DLL load failed error. Here is the code:
'''
     import torch
     import scipy
   import albumentations as A
  from ._nnls import nnls
  from albumentations.pytorch import ToTensorV2
  from tqdm import tqdm
  import torch.nn as nn
 import torch.optim as optim
 from unet_model import UNet
 from utilscar import (
load_checkpoint,
save_checkpoint,
get_loaders,
check_accuracy,
save_predictions_as_imgs,
)

  # Hyperparameters etc.
LEARNING_RATE = 1e-4
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
BATCH_SIZE = 16
NUM_EPOCHS = 3
NUM_WORKERS = 2
IMAGE_HEIGHT = 160  # 1280 originally
IMAGE_WIDTH = 240  # 1918 originally
PIN_MEMORY = True
LOAD_MODEL = False
TRAIN_IMG_DIR = "Dataset/train_images/"
TRAIN_MASK_DIR = "Dataset/train_masks/"
VAL_IMG_DIR = "Dataset/val_images/"
VAL_MASK_DIR = "Dataset/val_masks/"

def train_fn(loader, model, optimizer, loss_fn, scaler):
loop = tqdm(loader)

for batch_idx, (data, targets) in enumerate(loop):
    data = data.to(device=DEVICE)
    targets = targets.float().unsqueeze(1).to(device=DEVICE)

    # forward
    with torch.cuda.amp.autocast():
        predictions = model(data)
        loss = loss_fn(predictions, targets)

    # backward
    optimizer.zero_grad()
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

    # update tqdm loop
    loop.set_postfix(loss=loss.item())

def main():
   train_transform = A.Compose(
    [
        A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
        A.Rotate(limit=35, p=1.0),
        A.HorizontalFlip(p=0.5),
        A.VerticalFlip(p=0.1),
        A.Normalize(
            mean=[0.0, 0.0, 0.0],
            std=[1.0, 1.0, 1.0],
            max_pixel_value=255.0,
        ),
        ToTensorV2(),
    ],
)

val_transform = A.Compose(
    [
        A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
        A.Normalize(
            mean=[0.0, 0.0, 0.0],
            std=[1.0, 1.0, 1.0],
            max_pixel_value=255.0,
        ),
        ToTensorV2(),
    ],
)

model = UNet(in_channels=3, out_channels=1).to(DEVICE)
loss_fn = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)

train_loader, val_loader = get_loaders(
    TRAIN_IMG_DIR,
    TRAIN_MASK_DIR,
    VAL_IMG_DIR,
    VAL_MASK_DIR,
    BATCH_SIZE,
    train_transform,
    val_transform,
    NUM_WORKERS,
    PIN_MEMORY,
)

if LOAD_MODEL:
    load_checkpoint(torch.load("my_checkpoint.pth.tar"), model)

check_accuracy(val_loader, model, device=DEVICE)
scaler = torch.cuda.amp.GradScaler()

for epoch in range(NUM_EPOCHS):
    train_fn(train_loader, model, optimizer, loss_fn, scaler)

    # save model
    checkpoint = {
        "state_dict": model.state_dict(),
        "optimizer":optimizer.state_dict(),
    }
    save_checkpoint(checkpoint)

    # check accuracy
    check_accuracy(val_loader, model, device=DEVICE)

    # print some examples to a folder
    save_predictions_as_imgs(
        val_loader, model, folder="saved_images/", device=DEVICE
    )

   if __name__ == "__main__":
   main()

'''
I found this error:
from ._nnls import nnls
ImportError: DLL load failed: The specified module could not be found.
Here I want to mention that, my python is 64 bit and all the libraries are also in 64 bit. I updated everything using conda update.

Output:
The below solution worked for me.

conda remove --force numpy, scipy

pip install -U numpy, scipy

Successfully installed numpy-1.19.5 scipy-1.5.4
Reference: https://github.com/conda/conda/issues/6396#issuecomment-350254762


Instruction: sqrt_vml_cpu not implemented for 'Long'

Input:
a = torch.full([2, 2], 9)

b = a.sqrt()
print(b)

b = a.rsqrt()
print(b)

RuntimeError: sqrt_vml_cpu not implemented for 'Long'

a is torch.LongTensor, but sqrt and rsqrt do not suppor Long, what should I do?

Output:
I couldn't reproduce your code because, when I run it, I get the following error (PyTorch 1.6):
RuntimeError: Providing a bool or integral fill value without setting the
optional `dtype` or `out` arguments is currently unsupported. In PyTorch 1.7,
when `dtype` and `out` are not set a bool fill value will return a tensor of
torch.bool dtype, and an integral fill value will return a tensor of
torch.long dtype.

However, I think your problem occurs because PyTorch does not support square root operation for Long/torch.int64. You should use another data type. This should do it:
a = torch.full([2, 2], 9, dtype=torch.float32)
b = a.sqrt()
b = a.rsqrt()


Instruction: Why is my parameter not changing and its gradient 0?

Input:
I am building a really simple model to learn the parameter in a Poisson model and I am not sure where I am going wrong. I am using pytorch.nn and doing the following.
I made some really simple fake data
# This is the value I am trying to estimate

x = torch.tensor(2.0)

# This is a value drawn from the Poisson(x) distribution
# In this example it is 4

y = torch.poisson(x).reshape(1)

Then I just set up a really simple model
# I initialised the parameter that is going to estimate x with a random value (0.2)
# and set that it requires a gradient

a = torch.tensor([0.2], requires_grad = True)

# I define the loss function with log_input set to false

loss_function = torch.nn.PoissonNLLLoss(log_input = False)

# Defined the model

def model(a):
    return torch.poisson(a)

# And the parameter to be optimised
# I chose SGD arbitrarily, maybe this is the problem?

optimizer = torch.optim.SGD([a], lr = 0.1)

Then I do iterations to update a
for i in range(2000):

    # Forward pass

    y_pred = model(a)

    # Compute the loss

    loss = loss_function(y_pred, y)

    # Backprop

    optimizer.zero_grad()

    loss.backward()

    # Update parameters

    optimizer.step()

The problem is after this the a is still 0.2 and if I call a.grad it is 0. Where am I going wrong?
Thanks in advance
UPDATE
I have tried instead to initiate a class for the model inheriting a nn.Module. However the same problem persists :
class learning_model(nn.Module):

    def __init__(self):
        super().__init__()
        self.a = nn.Parameter(torch.rand(1))
        self.a.requires_grad = True

    def forward(self):
        return torch.poisson(self.a)

model = learning_model()

loss_function = nn.PoissonNLLLoss(log_input = False)

optimizer = torch.optim.SGD(model.parameters(), lr = 0.1)

print(model.a)

Outputs:
Parameter containing:
tensor([0.1402], requires_grad=True)

Then:
for i in range(20):

    # Forward pass
    y_pred = model()

    # Compute the loss
    loss = loss_function(y_pred, y)

    # Backprop
    optimizer.zero_grad()

    loss.backward()

    # Update parameters
    optimizer.step()

print(model.a, '\n gradient:', model.a.grad)

Outputs:
Parameter containing:
tensor([0.1402], requires_grad=True)
 gradient: tensor([0.])

Output:
Your model doesn't have any trainable parameters. See this link
  torch.nn.parameter.Parameter
        A kind of Tensor that is to be considered a module parameter.


Instruction: Floating point operations results are different in Android, TensorFlow and Pytorch

Input:
I am trying to compare the floating point numbers on Android, Tensorflow and Pytorch. What I have observed is I am getting same result for Tensorflow and Android but different on Pytorch as Android and Tensorflow are performing round-down operation. Please see the following result:
TensorFlow
import tensorflow as tf
a=tf.convert_to_tensor(np.array([0.9764764, 0.79078835, 0.93181187]), dtype=tf.float32)

session = tf.Session()
result = session.run(a*a*a*a)

print(result)

PyTorch
import torch as th

th.set_printoptions(precision=8)

a=th.from_numpy(np.array([0.9764764, 0.79078835, 0.93181187])).type(th.FloatTensor)

result = a*a*a*a

print(result)

Android:
for (index in 0 until a.size) {
   var res = a[index] * a[index] * a[index] * a[index]
   result.add(res)
}

print("r=$result")

The result is as follows:
Android:           [0.9091739,  0.3910579,  0.7538986]
TensorFlow:        [0.9091739,  0.3910579,  0.7538986]
PyTorch:           [0.90917391, 0.39105791, 0.75389862]

You can see that PyTorch value is different. I know that this effect is minimal in this example but when we are performing training, and we are running for 1000 rounds with different batches and epochs, this different can be accumulated and can show un-desirable results. Can anyone point out how can we fix to have same number on three platforms.
Thanks.

Output:
You are not using the same level of precision when printing, hence why you get different results. Internally, those results are identical, it's just an artifact that you see due do the default of python to print only 7 digits after the comma.
If we set the same level of precision in numpy as the one you set in PyTorch, we get:
import numpy as np
import tensorflow as tf

# setting the print precision of numpy to 8 like in your pytorch example
np.set_printoptions(precision=8, floatmode="fixed")

a=tf.convert_to_tensor(np.array([0.9764764, 0.79078835, 0.93181187]), dtype=tf.float32)

session = tf.Session()
result = session.run(a*a*a*a)
print(result)

Results in:
[0.90917391 0.39105791 0.75389862]

Exactly the same as in PyTorch.


Instruction: Why embed dimemsion must be divisible by num of heads in MultiheadAttention?

Input:
I am learning the Transformer. Here is the pytorch document for MultiheadAttention. In their implementation, I saw there is a constraint:
 assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"

Why require the constraint: embed_dim must be divisible by num_heads?  If we go back to the equation

Assume:
Q, K,V are n x emded_dim matrices; all the weight matrices W is emded_dim x head_dim,
Then, the concat [head_i, ..., head_h] will be a n x (num_heads*head_dim) matrix;
W^O with size (num_heads*head_dim) x embed_dim
[head_i, ..., head_h] * W^O will become a n x embed_dim output
I don't know why we require embed_dim must be divisible by num_heads.
Let say we have num_heads=10000, the resuts are the same, since the matrix-matrix product will absort this information.

Output: When you have a sequence of seq_len x emb_dim (ie. 20 x 8) and you want to use num_heads=2, the sequence will be split along the emb_dim dimension. Therefore you get two 20 x 4 sequences. You want every head to have the same shape and if emb_dim isn't divisible by num_heads this wont work. Take for example a sequence 20 x 9 and again num_heads=2. Then you would get 20 x 4 and 20 x 5 which are not the same dimension.


Instruction: torch.nn.fucntional.interpolate(): Parameters Settings

Input:
I'm using torch.nn.functional.interpolate() to resize an image.
Firstly I use transforms.ToTensor() to transform an image into a tensor, which have size of (3, 252, 252), (252, 252) is the size of the imported image. What I want to do is to create a tensor with size (3, 504, 504) with interpolate() function.
I set the para scale_factor=2, but it returned a (3, 252, 504) tensor. Then I set it like scale_factor=(1,2,2) and received an error of dimensional conflict like this:
size shape must match input shape. Input is 1D, size is 3
So what way should I do to set the parameters in order to receive (3, 504, 504) tensor?

Output:
If you're using scale_factor you need to give batch of images not single image. So you need to add one batch by using unsqueeze(0) then give it to interpolate function as follows:
import torch
import torch.nn.functional as F

img = torch.randn(3, 252, 252)  # torch.Size([3, 252, 252])
img = img.unsqueeze(0)  # torch.Size([1, 3, 252, 252])

out = F.interpolate(img, scale_factor=(2, 2), mode='nearest')
print(out.size()) # torch.Size([1, 3, 504, 504])


Instruction: pytorch int32 to int64 conversion

Input:
I'm trying to convert a simple image mask to int64
image = np.array([[1, 2], [3, 4]], dtype='int32')

transform = Compose([
    torch.from_numpy,
    ConvertImageDtype(torch.int64)
])

However, transform(image) yields
tensor([[ 4294967296,  8589934592],
        [12884901888, 17179869184]])

Is there something wrong, or am I fundamentally misunderstanding something about how the conversion should work?

Output:
If you skip torch's conversion, the image is transformed correctly.
image = np.array([[1, 2], [3, 4]], dtype='int64')

transform = Compose([
    torch.from_numpy
])
transform(image)
# tensor([[1, 2],
#        [3, 4]])


Instruction: Importing MNIST dataset from local directory in a closed system

Input:
I am trying to run a tutorial based on MNIST data in a cluster and the node where training script runs don't have internet access so I am manually placing the MNIST dataset in the desired directory but I am getting Dataset not found error.
I am trying to run this tutorial on the cluster.
I have tried this answer but the answer doesn't resolve my problem.
Below is my code modifications -
import horovod.torch as hvd
train_dataset = \
datasets.MNIST('/scratch/netra/MNIST/processed/training.pt-%d' % hvd.rank(), train=True, download=True,
               transform=transforms.Compose([
                   transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))
               ]))
test_dataset = \
datasets.MNIST('/scratch/netra/MNIST/processed/test.pt-%d' % hvd.rank(), train=False,
               transform=transforms.Compose([
                   transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))
               ]))

How to resolve it?

Output:
You have to specify a root folder, not a full path to the processed file:

root (string): Root directory of dataset where MNIST/processed/training.pt
and  MNIST/processed/test.pt exist.

In your case:
root is /scratch/netra
Thus,
train_dataset = \
datasets.MNIST('/scratch/netra-%d' % hvd.rank(), train=True, download=True,
               transform=transforms.Compose([
                   transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))
               ]))
test_dataset = \
datasets.MNIST('/scratch/netra-%d' % hvd.rank(), train=False,
               transform=transforms.Compose([
                   transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))
               ]))


Instruction: how can i reshape these images as a 2d images tensors?

Input:
I am currently working with rgb images loaded as tensors and i would like to reshape them to be 2d tensors to implement deep neural networks on them
the shape which I am currently working on is :
images.shape

torch.Size([32, 3, 244, 244])

I dont know how to deal with the last two fields and also how to flatten the 3 channels of colors

Output:
Your requirement is too hazy and it's unclear what you want to achieve with these images. Do they come with labels? If not, do you want to use an unsupervised method such as an autoencoder? Looking at the shape of your images tensor:
torch.Size([32, 3, 244, 244])

This means that there are 32 color (RGB) images in this tensor. If your definition of 2D means converting them to grayscale images, then you can use the torchvision library.
images = [torchvision.transforms.ToPILImage()(img) for img in images]
images = [torchvision.transforms.Grayscale()(img) for img in images]

And to convert the PIL grayscale images back to torch tensor, use:
images = [torchvision.transforms.ToTensor()(img) for img in images]
images = torch.stack(images).to(device)

Now, the shape of images would be [32, 244, 244]
Flattening the much high resolution image at the very first layer is not a recommended idea. So, that's why you see in the computer vision literature that folks apply few convolution layers in the beginning of the model architecture so as to downsample them to smaller size (resolution) feature descriptors.


Instruction: Loss of Conv-neural-network not decreasing, instead obsoleting

Input:
I have a convolutional neural network in vgg architecture "style" (down below) to classify images if there is a cat on the picture, or a dog. My training set contains 25000 images cropped to 256px each side. I tried different learning rates, different loss functions and much more but my loss keeps fluctuating between 0.692 and 0.694, but it will not decrease...
normalize = transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
)

transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    normalize
])

# Output = [isDog, isCat]
train_data_list = []
train_data = []
target_list = []
plotlist = []
train_files = listdir("data/catsdogs/train/")

def loadTrainData():
    global train_data_list
    global train_data
    global target_list
    print("Loading data now...")

    amount = len(listdir("data/catsdogs/train/"))
    current = 0

    for i in range(amount):
        r = random.randint(0, len(train_files) - 1)
        file = train_files[r]
        train_files.remove(file)

        img = Image.open("data/catsdogs/train/" + file)
        img_tensor = transform(img)  # (3, 256, 256)

        isCat = 1 if 'cat' in file else 0
        isDog = 1 if 'dog' in file else 0
        target = [isCat, isDog]

        train_data_list.append(img_tensor)
        target_list.append(target)

        if len(train_data_list) >= 64:
            train_data.append((torch.stack(train_data_list), target_list))
            train_data_list = []
            target_list = []

        current = current + 1
        print("Loaded: {:.1f}%".format(current * 100 / amount))
    print("Loaded data successfully!")

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=1, padding=1)

        self.conv3 = nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1)
        self.conv4 = nn.Conv2d(32, 32, kernel_size=3, stride=1, padding=1)

        self.conv5 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1)
        self.conv6 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)

        self.dropout = nn.Dropout2d()
        self.relu = nn.ReLU(inplace=True)
        self.pool = nn.MaxPool2d(kernel_size=3, stride=3, padding=0, dilation=1, ceil_mode=False)

        self.fc1 = nn.Linear(5184, 1296)
        self.fc2 = nn.Linear(1296, 2)

    def forward(self, x):
        # Block 1
        x = self.conv1(x)
        x = self.relu(x)

        x = self.conv2(x)
        x = self.relu(x)

        x = self.pool(x)

        # Block 2
        x = self.conv3(x)
        x = self.relu(x)

        x = self.conv4(x)
        x = self.relu(x)

        x = self.pool(x)

        # Block 3
        x = self.conv5(x)
        x = self.relu(x)

        x = self.conv6(x)
        x = self.relu(x)

        x = self.pool(x)

        x = x.view(-1, 5184)

        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)

        return torch.sigmoid(x)

model = Network()
model = model.cuda()

optimizer = optim.SGD(model.parameters(), lr=0.0001, weight_decay=0.0016)

def train(epoch):
    global optimizer

    model.train()
    batch_id = 0
    for data, target in train_data:
        data = data.cuda()
        target = torch.Tensor(target).cuda()

        data = Variable(data)
        target = Variable(target)

        optimizer.zero_grad()

        out = model(data)
        criterion = F.binary_cross_entropy

        loss = criterion(out, target)
        loss.backward()

        optimizer.step()

        plotlist.append(loss)

        print('Train Epoch: {},  {:.0f}% ,\tLoss: {:.6f}'.format(
            epoch, 100. * batch_id / len(train_data), loss.item()
        ))
        batch_id = batch_id + 1

loadTrainData()

for epoch in range(25):
    train(epoch)

plt.plot(plotlist)
plt.show()
plt.ylabel("Loss")
plt.savefig("lossPlot.png")

Here is a plot of my loss over 5 iterations:

Also with a higher learning rate the fluctuation only gets stronger, for 0.1 lr between 0.5 and 0.7.

Output:
have you tried adding momentum to your SGD optimizer?
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.0016, momentum=0.9)

Or, a different optimizer such as Adam or AdaDelta, which will use adaptive learning rate?
Also, it does not look like your training data is shuffled - can it happen than some batches have all cats and some batches have all dogs, pulling the gradient descent in different in opposite directions every few steps? It may be better to shuffle your training data after every epoch and do the batching on top of that. torch.utils.data.DataLoader class may be of some help in this.
What's the naming scheme of your files? Do the variables 'isCat' and 'isDog' have correct values?
What happens when you try to train on only 100 examples - is your model able to learn the train data in this simple case - this should hopefully rule out some obvious bugs.


Instruction: RuntimeError: CUDA error: device-side assert triggered on loss function

Input:
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [54,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [55,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [56,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [57,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [58,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [59,0,0] Assertion input_val >= zero && input_val <= one failed.
Traceback (most recent call last):
File "run_toys.py", line 215, in
loss = criterion(torch.reshape(out, [-1, dataset.out_dim]), torch.reshape(target, [-1, dataset.out_dim]))
File "/usr/local/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/python3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 530, in forward
return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
File "/usr/local/python3/lib/python3.6/site-packages/torch/nn/functional.py", line 2526, in
binary_cross_entropy
input, target, weight, reduction_enum)
RuntimeError: CUDA error: device-side assert triggered
The Code
criterion = nn.CrossEntropyLoss()
loss = criterion(torch.reshape(out, [-1, dataset.out_dim]), torch.reshape(target, [-1, dataset.out_dim]))
loss = torch.mean(loss)

The shape of the target and output is the same # torch.Size([640, 32])
The model runs on my CPU OK, but running on GPU is the issue

Output:
There might be two reasons of the error:

As the log says input_val is not between the range [0; 1]. So you should ensure that model outputs are in that range. You can use torch.clamp() of pytorch. Before calculating the loss add the following line:

    out = out.clamp(0, 1)

Maybe you are sure that model outputs are in the range [0; 1]. Then very common problem is output contains some nan values which triggers assert as well. To prevent this you can use the following trick, again before calculating the loss:

    out[out!=out] = 0 # or 1 depending on your model's need

Here the trick is using nan!=nan property, we should change them to some valid number.


Instruction: In training mode, targets should be passed

Input:
I am new to deep learning and have the project on detecting traffic lights in university where we can use open-source code.
So, I tried to run the code on kaggle https://www.kaggle.com/endoruk1234/trafficlightdetection-fasterrcnn-pytorch/log
However on the stage of testing the saved model on a video, I got this mistake 'In training mode, targets should be passed'.
I am not sure why do I need to pass targets on the testing stage. I don't understand is it a problem with the initial model, or video capture part is written with mistakes.
The model

    from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
    def _get_instance_segmentation_model(num_classes):
        model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
        in_features = model.roi_heads.box_predictor.cls_score.in_features
        model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
        return model
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
    N_CLASS = 4
    INP_FEATURES = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(INP_FEATURES, N_CLASS)
    model.to(device)
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.Adam(params)
    lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer)

Training

    lossHist = LossAverager()
    valLossHist = LossAverager()

    for epoch in range(EPOCHS):

        start_time = time()
        model.train()
        lossHist.reset()

        for images, targets, image_ids in tqdm(trainDataLoader):
            #bbox = check_bbox(bbox)
            images = torch.stack(images).to(device)
            targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

            bs = images.shape[0]

            loss_dict = model(images, targets)

            totalLoss = sum(loss for loss in loss_dict.values())
            lossValue = totalLoss.item()

            lossHist.update(lossValue,bs)

            optimizer.zero_grad()
            totalLoss.backward()
            optimizer.step()

        if lr_scheduler is not None:
            lr_scheduler.step(totalLoss)

        print(f"[{str(datetime.timedelta(seconds = time() - start_time))[2:7]}]")
        print(f"Epoch {epoch}/{EPOCHS}")
        print(f"Train loss: {lossHist.avg}")

        if(epoch == 10):
            torch.save(model.state_dict(), 'fasterrcnn_resnet{}_fpn.pth'.format(epoch))

    torch.save(model.state_dict(), 'fasterrcnn_resnet{}_fpn.pth'.format(epoch))

Testing on a video with a mistake

    while(True):
        ret, input = cap.read()
        image = input.copy()
        input = preprocess(input).float()
        input = input.unsqueeze_(0)
        input = input.type(torch.cuda.FloatTensor)

        print(input)

        result = model(input)

        boxes = result[0]['boxes'].type(torch.cuda.FloatTensor)
        scores = result[0]['scores'].type(torch.cuda.FloatTensor)
        labels = result[0]['labels'].type(torch.cuda.FloatTensor)

        mask = nms(boxes,scores,0.3)
        boxes = boxes[mask]
        scores = scores[mask]
        labels = labels[mask]

        boxes = boxes.data.cpu().numpy().astype(np.int32)
        scores = scores.data.cpu().numpy()
        labels = labels.data.cpu().numpy()

        mask = scores >= 0.5
        boxes = boxes[mask]
        scores = scores[mask]
        labels = labels[mask]

        colors = {1:(0,255,0), 2:(255,255,0), 3:(255,0,0)}

        for box,label in zip(boxes,labels):
            image = cv2.rectangle(image,
                              (box[0], box[1]),
                              (box[2], box[3]),
                              (0,0,255), 1)

        cv2.imshow("image", image)

        if cv2.waitKey(0):
            break

    ValueError                                Traceback (most recent call last)
    <ipython-input-84-e32f9d25d942> in <module>()
          8     print(input)
          9
    ---> 10     result = model(input)
         11
         12     boxes = result[0]['boxes'].type(torch.cuda.FloatTensor)

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
        725             result = self._slow_forward(*input, **kwargs)
        726         else:
    --> 727             result = self.forward(*input, **kwargs)
        728         for hook in itertools.chain(
        729                 _global_forward_hooks.values(),

    /usr/local/lib/python3.7/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
         58         """
         59         if self.training and targets is None:
    ---> 60             raise ValueError("In training mode, targets should be passed")
         61         if self.training:
         62             assert targets is not None

    ValueError: In training mode, targets should be passed

Thank you in advance! If you can tell me how to correct the model or videocapture code I will be very grateful.

Output:
In your example you don't explain how you load your model, but I think you have forgotten model.eval(). This function  is a kind of switch for some specific layers/parts of the model that behave differently during training and inference (evaluating) time.
To make inferences, you can load your model like this :
model.load_state_dict(torch.load("/content/gdrive/MyDrive/Models/model_Resnet.pth"))
model.eval()


Instruction: Why does it take so long print the value of a GPU tensor in Pytorch?

Input:
I wrote this pytorch program to compute a 5000*5000 matrix multiplication on GPU, 100 iterations.
import torch
import numpy as np
import time

N = 5000
x1 = np.random.rand(N, N)

######## a 5000*5000 matrix multiplication on GPU, 100 iterations #######
x2 = torch.tensor(x1, dtype=torch.float32).to("cuda:0")

start_time = time.time()
for n in range(100):
    G2 = x2.t() @ x2
print(G2.size())
print("It takes", time.time() - start_time, "seconds to compute")
print("G2.device:", G2.device)

start_time2 = time.time()
# G4 = torch.zeros((5,5),device="cuda:0")
G4 = G2[:5, :5]
print("G4.device:", G4.device)
print("G4======", G4)
# G5=G4.cpu()
# print("G5.device:",G5.device)
print("It takes", time.time() - start_time2, "seconds to transfer or display")

Here is the result on my laptop:

torch.Size([5000, 5000])

It takes 0.22243595123291016 seconds to compute

G2.device: cuda:0

G4.device: cuda:0

G4====== tensor([[1636.3195, 1227.1913, 1252.6871, 1242.4584, 1235.8160],
[1227.1913, 1653.0522, 1260.2621, 1246.9526, 1250.2871],
[1252.6871, 1260.2621, 1685.1147, 1257.2373, 1266.2213],
[1242.4584, 1246.9526, 1257.2373, 1660.5951, 1239.5414],
[1235.8160, 1250.2871, 1266.2213, 1239.5414, 1670.0034]],
device='cuda:0')

It takes 60.13639569282532 seconds to transfer or display
Process finished with exit code 0

I am confused why it takes so much time to display the variable G5 on GPU, since is only 5*5 in size.
BTW, I use "G5=G4.cpu()" to transfer the variable on GPU to CPU, it takes so much time too.
My develop enviroment (rather old laptop):

pytorch 1.0.0

CUDA 8.0

Nvidia GeForce GT 730m

Windows 10 Professional

when increasing iteration times, the compute time do not increase obviously, but the transfer or display increased obviously, Why? Can somebody interpret, thanks so much.

Output:
Pytorch CUDA operations are asynchronous. Most operations on GPU tensors are actually non blocking until a derived result is requested. This means that until you ask for a CPU version of a tensor, commands like matrix multiply are basically being processed in parallel to your code. When you stop the timer there's no guarantee that the operation has been completed. You can read more about this in the docs.
To time chunks of your code properly you should add calls to torch.cuda.synchronize. This function should be called twice, once right before you start your timer, and once right before you stop your timer. Outside of profiling your code you should avoid calls to this function as it may slow down overall performance.


Instruction: 'LSTM' object has no attribute '_flat_weights_names'

Input:
While executing iNltk library, I am getting an error. I have latest versions of pytorch and torchvision.
'LSTM' object has no attribute '_flat_weights_names'

After re-searching on some blogs some people suggested to downgrade the version to 1.2 So i tried below installation from https://pytorch.org/get-started/previous-versions/
pip install torch==1.2.0+cu92 torchvision==0.4.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

However, getting errors
ERROR: Could not find a version that satisfies the requirement torch==1.2.0+cpu
ERROR: No matching distribution found for torch==1.2.0+cpu

Also, 1.3.1 version is missing.
Anybody has any idea about how to downgrade to 1.3.1 or 1.2.0?
Thanks in advance
PD

Output:
I tried using pip, but that did not work for me. conda fixed the issue.
Setup a conda environment first and activate it. Install iNLTK using pip in conda as follows:
pip install inltk

Remove the version of PyTorch installed as a dependency for iNLTK.
pip uninstall torch

Install the desired version of PyTorch.
conda install pytorch==1.3.0 -c pytorch

Refer: Previous PyTorch Versions