Instruction: Using self in init part of a class in Python

Input:
Is there any difference between the following two codes related to initializing a class in Python?
class summation:
    def __init__(self, f, s):
        self.first = f
        self.second = s
        self.summ = self.first + self.second
    .
    .
    .

class summation:
    def __init__(self, f, s):
        self.first = f
        self.second = s
        self.summ = f + s
    .
    .
    .

If there exists any difference, what is that, and which code is preferable?
Edit: I am going to write an artificial neural network with Python (and Pytorch). In fact, the above two codes are just some examples. In the actual case, I have seen in various resources that when there exists self.input = input in the initialization of a class, in other parts it is used as self.input, not input.
My questions: What are the differences between these two approaches? Why is the use of self.input preferable, in my case?
Example: (from https://docs.dgl.ai/en/latest/tutorials/models/1_gnn/4_rgcn.html#sphx-glr-tutorials-models-1-gnn-4-rgcn-py)
import torch
import torch.nn as nn
import torch.nn.functional as F
from dgl import DGLGraph
import dgl.function as fn
from functools import partial

class RGCNLayer(nn.Module):
    def __init__(self, in_feat, out_feat, num_rels, num_bases=-1, bias=None,
                 activation=None, is_input_layer=False):
        super(RGCNLayer, self).__init__()
        self.in_feat = in_feat
        self.out_feat = out_feat
        self.num_rels = num_rels
        self.num_bases = num_bases
        self.bias = bias
        self.activation = activation
        self.is_input_layer = is_input_layer

        # sanity check
        if self.num_bases <= 0 or self.num_bases > self.num_rels:
            self.num_bases = self.num_rels

        # weight bases in equation (3)
        self.weight = nn.Parameter(torch.Tensor(self.num_bases, self.in_feat,
                                                self.out_feat))
        if self.num_bases < self.num_rels:
            # linear combination coefficients in equation (3)
            self.w_comp = nn.Parameter(torch.Tensor(self.num_rels, self.num_bases))

        # add bias
        if self.bias:
            self.bias = nn.Parameter(torch.Tensor(out_feat))

        # init trainable parameters
        nn.init.xavier_uniform_(self.weight,
                                gain=nn.init.calculate_gain('relu'))
        if self.num_bases < self.num_rels:
            nn.init.xavier_uniform_(self.w_comp,
                                    gain=nn.init.calculate_gain('relu'))
        if self.bias:
            nn.init.xavier_uniform_(self.bias,
                                    gain=nn.init.calculate_gain('relu'))

    def forward(self, g):
        if self.num_bases < self.num_rels:
            # generate all weights from bases (equation (3))
            weight = self.weight.view(self.in_feat, self.num_bases, self.out_feat)
            weight = torch.matmul(self.w_comp, weight).view(self.num_rels,
                                                        self.in_feat, self.out_feat)
        else:
            weight = self.weight

        if self.is_input_layer:
            def message_func(edges):
                # for input layer, matrix multiply can be converted to be
                # an embedding lookup using source node id
                embed = weight.view(-1, self.out_feat)
                index = edges.data['rel_type'] * self.in_feat + edges.src['id']
                return {'msg': embed[index] * edges.data['norm']}
        else:
            def message_func(edges):
                w = weight[edges.data['rel_type']]
                msg = torch.bmm(edges.src['h'].unsqueeze(1), w).squeeze()
                msg = msg * edges.data['norm']
                return {'msg': msg}

        def apply_func(nodes):
            h = nodes.data['h']
            if self.bias:
                h = h + self.bias
            if self.activation:
                h = self.activation(h)
            return {'h': h}

        g.update_all(message_func, fn.sum(msg='msg', out='h'), apply_func)

Output:
No. there is no difference between these two approaches in your case with this level of information. but could they? Yes. they could. if they have some modifications in their setters or getters. later in my answer I'll show you how.
First of all, I prefer using this one:
class summation:
    def __init__(self, f, s):
        self.first = f
        self.second = s
    @property
    def summ(self):
        return self.first+self.second

the above implementation calculates the summation on demand. so when you change self.first or self.second, summ will be calculated automatically. you can access the sum as you did before.
s = summation(1,9)
print(s.summ)
# 10
s.first = 2
s.second = 3
print(s.summ)
# 5

So, How could they be different?
let's implements them as follows. in setters I doubled the inputs to show you how setters can affect the results. it's just an imaginary example and is not exactly what you wrote.
class summation1:
    def __init__(self, f, s):
        self.first = f
        self.second = s
        self.summ = self.first + self.second

    @property
    def first(self):
      return self.__first

    @first.setter
    def first(self,f):
      self.__first = f*2

    @property
    def second(self):
      return self.__second

    @second.setter
    def second(self,s):
      self.__second = s*2

class summation2:
    def __init__(self, f, s):
        self.first = f
        self.second = s
        self.summ = f + s

    @property
    def first(self):
      return self.__first

    @first.setter
    def first(self,f):
      self.__first = f*2

    @property
    def second(self):
      return self.__second

    @second.setter
    def second(self,s):
      self.__second = s*2

now let's take a look at the outputs:
a = 3
b = 2
s1 = summation1(a,b)
s2 = summation2(a,b)

print(s1.summ)
# 10
print(s2.summ)
# 5

so, if you are not sure what to choose between those two, maybe the first approach is what you need.


Instruction: problem with nn.ModuleDict ('method' object is not subscriptable)

Input:
I am trying to use nn.ModuleDict following this documentation page:
I have this PyTorch network:
class Net(nn.Module):
    def __init__(self, kernel_size):
        super(Net, self).__init__()
        modules = {}
        modules["layer1"] = nn.Conv2d(3, 16, kernel_size=kernel_size, stride=1, padding=2)
        self.modules = nn.ModuleDict(modules)

    def forward(self, x):
        x = self.modules["layer1"](x)

when I use the forward method, I get the following error:

'method' object is not subscriptable

when I change the forward method to:
  def forward(self, x):
        x = self.modules()["layer1"](x)

I get the following error:

TypeError: 'generator' object is not subscriptable

Output:
The key "modules" is already used by nn.Module. The property is used to retrieve all modules of the model: see nn.Module.modules. You need to use another property name.
For example:
class Net(nn.Module):
    def __init__(self, kernel_size):
        super(Net, self).__init__()
        modules = {}
        modules["layer1"] = nn.Conv2d(3, 16,
            kernel_size=kernel_size, stride=1, padding=2)
        self.layers = nn.ModuleDict(modules)

    def forward(self, x):
        x = self.layers["layer1"](x)
        return x


Instruction: PyTorch model + HTTP API = very slow execution

Input:
I have:

ML model (PyTorch) that vectorizes data and makes a prediction in ~3.5ms (median ≈ mean)
HTTP API (FastAPI + uvicorn) that serves simple requests in ~2ms

But when I combine them, the median response time becomes almost 200ms.
What can be the reason for such degradation?

Note that:

I also tried aiohttp alone, aiohttp + gunicorn and Flask development server for serving - same result
I tried to send 2, 20 and 100 requests per second - same result
I do realize that parallel requests can lead to decreased latency, but not 30 times!
CPU load is only ~7%

Here's how I measured model performance (I measured the median time separately, it's nearly the same as the mean time):
def predict_all(predictor, data):
    for i in range(len(data)):
        predictor(data[i])

data = load_random_data()
predictor = load_predictor()
%timeit predict_all(predictor, data)
# manually divide total time by number of records in data

Here's FastAPI version:
from fastapi import FastAPI
from starlette.requests import Request
from my_code import load_predictor

app = FastAPI()

app.predictor = load_predictor()

@app.post("/")
async def root(request: Request):
    predictor = request.app.predictor
    data = await request.json()
    return predictor(data)

HTTP performance test:
wrk2 -t2 -c50 -d30s -R100 --latency -s post.lua http://localhost:8000/

EDIT.
Here's a slightly modified version which I tried with and without async:
@app.post("/")
# async def root(request: Request, user_dict: dict):
def root(request: Request, user_dict: dict):
    predictor = request.app.predictor
    start_time = time.time()
    y = predictor(user_dict)
    finish_time = time.time()
    logging.info(f"user {user_dict['user_id']}: "
                 "prediction made in {:.2f}ms".format((finish_time - start_time) * 1000))
    return y

So I just added logging of prediction time.
Log for async version:
2021-02-03 11:14:31,822: user 12345678-1234-1234-1234-123456789123: prediction made in 2.87ms
INFO:     127.0.0.1:49284 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,329: user 12345678-1234-1234-1234-123456789123: prediction made in 3.93ms
INFO:     127.0.0.1:49286 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,345: user 12345678-1234-1234-1234-123456789123: prediction made in 15.06ms
INFO:     127.0.0.1:49287 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,351: user 12345678-1234-1234-1234-123456789123: prediction made in 4.78ms
INFO:     127.0.0.1:49288 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,358: user 12345678-1234-1234-1234-123456789123: prediction made in 6.85ms
INFO:     127.0.0.1:49289 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,363: user 12345678-1234-1234-1234-123456789123: prediction made in 3.71ms
INFO:     127.0.0.1:49290 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,369: user 12345678-1234-1234-1234-123456789123: prediction made in 5.49ms
INFO:     127.0.0.1:49291 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,374: user 12345678-1234-1234-1234-123456789123: prediction made in 5.00ms

So prediction is fast, less than 10ms on average, but whole request takes 200ms.
Log for sync version:
2021-02-03 11:17:58,332: user 12345678-1234-1234-1234-123456789123: prediction made in 65.49ms
2021-02-03 11:17:58,334: user 12345678-1234-1234-1234-123456789123: prediction made in 23.05ms
INFO:     127.0.0.1:49481 - "POST / HTTP/1.1" 200 OK
INFO:     127.0.0.1:49482 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:17:58,338: user 12345678-1234-1234-1234-123456789123: prediction made in 72.39ms
2021-02-03 11:17:58,341: user 12345678-1234-1234-1234-123456789123: prediction made in 78.66ms
2021-02-03 11:17:58,341: user 12345678-1234-1234-1234-123456789123: prediction made in 85.74ms

Now prediction takes long! For whatever reason, exactly the same call, but made in synchronous context, started to take ~30 times longer. But the whole request takes approximately the same time - 160-200ms.

Output:
In endpoints that does highly intensive calculations and which presumably takes longer when compared to the other endpoints, use a non-coroutine handler.
When you use def instead of async def, by default FastAPI will use run_in_threadpool from Starlette and which also uses loop.run_in_executor underneath.
run_in_executor will execute the function in the default loops executor, it executes the function in a seperate thread, also you might want to check options like ProcessPoolExecutor and ThreadPoolExecutor if you are doing highly CPU intensive work.
This math simple math helps a lot when working with coroutines.
function
   if function_takes ≥ 500ms
       use `def`
   else
       use `async def`

Making your function non-coroutine should do good.
@app.post("/")
def root(request: Request):
    predictor = request.app.predictor
    data = await request.json()
    return predictor(data)


Instruction: Pytorch:RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Input:
I set my model and data to the same device,
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net.to(device)

and I also do this:
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)

but the error still exists. When it comes to 5000 times or more, the error will take place.

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

The following is the whole training code, I hope you can answer it. Thanks!
import torch
import os
import torchvision.transforms as transforms
from PIL import Image
from torch import nn
from torch.utils.data import Dataset, DataLoader

captcha_list = list('0123456789abcdefghijklmnopqrstuvwxyz_')
captcha_length = 6

# 验证码文本转为向量
def text2vec(text):
    vector = torch.zeros((captcha_length, len(captcha_list)))
    text_len = len(text)
    if text_len > captcha_length:
        raise ValueError("验证码超过6位啦！")
    for i in range(text_len):
        vector[i,captcha_list.index(text[i])] = 1
    return vector

# 验证码向量转为文本
def vec2text(vec):
    label = torch.nn.functional.softmax(vec, dim =1)
    vec = torch.argmax(label, dim=1)
    for v in vec:
        text_list = [captcha_list[v] for v in vec]
    return ''.join(text_list)

# 加载所有图片，并将验证码向量化
def make_dataset(data_path):
    img_names = os.listdir(data_path)
    samples = []
    for img_name in img_names:
        img_path = data_path+img_name
        target_str = img_name.split('_')[0].lower()
        samples.append((img_path, target_str))
    return samples

class CaptchaData(Dataset):
    def __init__(self, data_path, transform=None):
        super(Dataset, self).__init__()
        self.transform = transform
        self.samples = make_dataset(data_path)

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, index):
        img_path, target = self.samples[index]
        target = text2vec(target)
        target = target.view(1, -1)[0]
        img = Image.open(img_path)
        img = img.resize((140,44))
        img = img.convert('RGB') # img转成向量
        if self.transform is not None:
            img = self.transform(img)
        return img, target
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 第一层神经网络
        # nn.Sequential: 将里面的模块依次加入到神经网络中
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, padding=1), # 3通道变成16通道，图片：44*140
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(2)  # 图片：22*70
        )
        # 第2层神经网络
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 64, kernel_size=3), # 16通道变成64通道，图片：20*68
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2)  # 图片：10*34
        )
        # 第3层神经网络
        self.layer3 = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3), # 16通道变成64通道，图片：8*32
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(2)  # 图片：4*16
        )
        # 第4层神经网络
        self.fc1 = nn.Sequential(
            nn.Linear(4*16*128, 1024),
            nn.Dropout(0.2),  # drop 20% of the neuron
            nn.ReLU()
        )
        # 第5层神经网络
        self.fc2 = nn.Linear(1024, 6*37) # 6:验证码的长度， 37: 字母列表的长度
    #前向传播
    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        return x
net = Net()

def calculat_acc(output, target):
    output, target = output.view(-1, len(captcha_list)), target.view(-1, len(captcha_list)) # 每37个就是一个字符
    output = nn.functional.softmax(output, dim=1)
    output = torch.argmax(output, dim=1)
    target = torch.argmax(target, dim=1)
    output, target = output.view(-1, captcha_length), target.view(-1, captcha_length) #每6个字符是一个验证码
    c = 0
    for i, j in zip(target, output):
        if torch.equal(i, j):
            c += 1
    acc = c / output.size()[0] * 100
    return acc

def train(epoch_nums):
    # 数据准备
    transform = transforms.Compose([transforms.ToTensor()]) # 不做数据增强和标准化了
    train_dataset = CaptchaData('./sougou_com_Trains/', transform=transform)
    train_data_loader = DataLoader(train_dataset, batch_size=32, num_workers=0, shuffle=True, drop_last=True)

    test_data = CaptchaData('./sougou_com_Trains/', transform=transform)
    test_data_loader = DataLoader(test_data, batch_size=128, num_workers=0, shuffle=True, drop_last=True)
    # 更换设备
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    print('当前设备是:',device)
    net.to(device)

    criterion = nn.MultiLabelSoftMarginLoss() # 损失函数
    optimizer = torch.optim.Adam(net.parameters(), lr=0.001) # 优化器

    # 加载模型
    model_path = './module_build/model.pth'
    if os.path.exists(model_path):
        print('开始加载模型')
        checkpoint = torch.load(model_path)
        net.load_state_dict(checkpoint['model_state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])

    # 开始训练
    i = 1
    for epoch in range(epoch_nums):
        running_loss = 0.0
        net.train() # 神经网络开启训练模式
        for data in train_data_loader:
            if i%100==0:
                print(i)
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device) #数据发送到指定设备
            #每次迭代都要把梯度置零
            optimizer.zero_grad()
            # 关键步骤
            # 前向传播
            outputs = net(inputs)
            # 计算误差
            loss = criterion(outputs, labels)
            # 后向传播
            loss.backward()
            # 优化参数
            optimizer.step()
            running_loss += loss.item()
            if i % 2000 == 0:
                acc = calculat_acc(outputs, labels)
                print('第%s次训练正确率: %.3f %%, loss: %.3f' % (i,acc,running_loss/2000))
                running_loss = 0
                # 保存模型
                torch.save({
                            'model_state_dict':net.state_dict(),
                            'optimizer_state_dict':optimizer.state_dict(),
                            },model_path)
            i += 1
        # 结束一个epoch,计算测试集的正确率
        net.eval() #测试模式
        with torch.no_grad():
            for inputs, labels in test_data_loader:
                outputs = net(inputs)
                acc = calculat_acc(outputs, labels)
                print('测试集正确率: %.3f %%' % (acc))
                break # 只测试一个batch

        # 每5个epoch 更新学习率
        if epoch % 5 == 4:
            for p in optimizer.param_groups:
                p['lr'] *= 0.9
train(10)

Output:
In evaluation part: Do this
net.eval() #测试模式
with torch.no_grad():
    for inputs, labels in test_data_loader:
        inputs, labels = inputs.to(device), labels.to(device)
        outputs = net(inputs)
        acc = calculat_acc(outputs, labels)
        print('测试集正确率: %.3f %%' % (acc))
        break # 只测试一个batch

This will work since you dint move your data from test loader to device.


Instruction: What do the charts in the System Panels signify in Wandb (PyTorch)

Input:
I recently started using the wandb module with my PyTorch script, to ensure that the GPU's are operating efficiently. However, I am unsure as to what exactly the charts indicate.
I have been following the tutorial in this link, https://lambdalabs.com/blog/weights-and-bias-gpu-cpu-utilization/ , and was confused by this plot:

I am uncertain about the GPU % and the GPU Memory Access % charts. The descriptions in the blog are as following:

GPU %: This graph is probably the most important one. It tracks the percent of the time over the past sample period during which one or more kernels was executing on the GPU. Basically, you want this to be close to 100%, which means GPU is busy all the time doing data crunching. The above diagram has two curves. This is because there are two GPUs and only of them (blue) is used for the experiment. The Blue GPU is about 90% busy, which means it is not too bad but still has some room for improvement. The reason for this suboptimal utilization is due to the small batch size (4) we used in this experiment. The GPU fetches a small amount of data from its memory very often, and can not saturate the memory bus nor the CUDA cores. Later we will see it is possible to bump up this number by merely increasing the batch size.

GPU Memory Access %: This is an interesting one. It measures the percent of the time over the past sample period during which GPU memory was being read or written. We should keep this percent low because you want GPU to spend most of the time on computing instead of fetching data from its memory. In the above figure, the busy GPU has around 85% uptime accessing memory. This is very high and caused some performance problem. One way to lower the percent here is to increase the batch size, so data fetching becomes more efficient.

I had the following questions:

The aforementioned values do not sum to 100%. It seems as though our GPU can either be spending time on computation or spending time on reading/writing memory. How can the sum of these two values be greater than 100%?
Why does increasing batch size decrease the time spent accessing GPU Memory?

Output:
GPU Utilization and GPU memory access should add up to 100% is true if the hardware is doing both the process sequentially. But modern hardware doesn't do operations like this. GPU will be busy computing the numbers at the same time it will be accessing the memory.

GPU% is actually GPU Utilization %. We want this to be 100%. Thus it will do the desired computation 100% of the time.
GPU memory access % is the amount of time GPU is either reading from or writing to GPU memory. We want this number to be low. If the GPU memory access % is high there can be some delay before GPU can use the data to compute on. That doesn't mean that it's a sequential process.

W&B allows you to monitor both the metrics and take decisions based on them. Recently I implemented a data pipeline using tf.data.Dataset. The GPU utilization was close to 0% while memory access was close to 0% as well. I was reading three different image files and stacking them. Here CPU was the bottleneck. To counter this I created a dataset by stacking the images. The ETA went from 1h per epoch to 3 min.

From the plot, you can infer that the memory access of GPU increased while GPU utilization is close to 100%. CPU utilization decreased which was the bottleneck.

Here's a nice article by Lukas answering this question.


Instruction: print the value of variables after each line of code

Input:
I'm trying to understand how the following PyTorch code works. To know how each function works and what they output & to know the outputted variables value and size, I'm using print() after each line of code.
s = pc()
for _ in trange(max_length):
    if self.onnx:
        outputs = torch.tensor(self.decoder_with_lm_head.run(None, {"input_ids": generated.cpu().numpy(),
                                        "encoder_hidden_states": encoder_outputs_prompt})[0][0])
        print(f'decoder output -- {outputs}')
    else:
        outputs = self.decoder_with_lm_head(input_ids=generated, encoder_hidden_states=encoder_outputs_prompt)[0]
    next_token_logits = outputs[-1, :] / (temperature if temperature > 0 else 1.0)
    print(f'next_token_logits -- {next_token_logits}')
    if int(next_token_logits.argmax()) == 1:
        print(f'next token logits argmax  -- {int(next_token_logits.argmax())}')
        break
    new_logits.append(next_token_logits)
    print(f'new_logits  { i }   -- {new_logits}')
    print(f'generated     -- {generated}')
    print(f'generated  view list   -- {set(generated.view(-1).tolist())}')
    for _ in set(generated.view(-1).tolist()):
        next_token_logits[_] /= repetition_penalty
        print(f'ext_token_logits[_]   -- {next_token_logits[_]}')
    if temperature == 0:  # greedy sampling:
        next_token = torch.argmax(next_token_logits).unsqueeze(0)
        print(f'next_token   --   {next_token}')
    else:
        filtered_logits = top_k_top_p_filtering(next_token_logits, top_k=top_k, top_p=top_p)
        next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=1)
    generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)
    print(f'generated end    --   {generated}')
    new_tokens = torch.cat((new_tokens, next_token), 0)
    print(f'new_tokens end  -- {new_tokens}')
    i += 1
    print(f'--------------------------\n')
e = pc()
ap = e-s
print(ap)
print(timedelta(ap))
return self.tokenizer.decode(new_tokens), new_logit

my question, is there a more efficient way of tracking these values & their shape, or are there any libraries that handle this task.

Output:
after doing a lot of searches I found a library that fits my requirements. The library is pysnooper, instead of using print() function after each line of code, now I can just use pysnooper's decorator
@pysnooper.snoop()
def greedy_search(input_text, num_beam, max_length,  max_context_length=512):
     ...

and it will print all the variable's values with their corresponding time of execution.
for more info refer to its GitHub page.
I feel like debuggers are a bit complicated & I also want to run it on notebooks. this is the simplest option I found. any suggestions are welcome.


Instruction: How to handle odd resolutions in Unet architecture PyTorch

Input:
I'm implementing a U-Net based architecture in PyTorch. At train time, I've patches of size 256x256 which doesn't cause any problem. However at test time, I've full HD images (1920x1080). This is causing a problem during skip connections.
Downsampling 1920x1080 3 times gives 240x135. If I downsample one more time, the resolution becomes 120x68 which when upsampled gives 240x136. Now, I cannot concatenate these two feature maps. How can I solve this?
PS: I thought this is a fairly common problem, but I didn't get any solution or even mentioning of this problem anywhere on the web. Am I missing something?

Output:
It is a very common problem in segmentation networks where skip-connections are often involved in the decoding process. Networks usually (depending on the actual architecture) require input size that has side lengths as integer multiples of the largest stride (8, 16, 32, etc.).
There are two main ways:

Resize input to the nearest feasible size.
Pad the input to the next larger feasible size.

I prefer (2) because (1) can cause small changes in the pixel level for all the pixels, leading to unnecessary blurriness. Note that we usually need to recover the original shape afterward in both methods.
My favorite code snippet for this task (symmetric padding for height/width):
import torch
import torch.nn.functional as F

def pad_to(x, stride):
    h, w = x.shape[-2:]

    if h % stride > 0:
        new_h = h + stride - h % stride
    else:
        new_h = h
    if w % stride > 0:
        new_w = w + stride - w % stride
    else:
        new_w = w
    lh, uh = int((new_h-h) / 2), int(new_h-h) - int((new_h-h) / 2)
    lw, uw = int((new_w-w) / 2), int(new_w-w) - int((new_w-w) / 2)
    pads = (lw, uw, lh, uh)

    # zero-padding by default.
    # See others at https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.pad
    out = F.pad(x, pads, "constant", 0)

    return out, pads

def unpad(x, pad):
    if pad[2]+pad[3] > 0:
        x = x[:,:,pad[2]:-pad[3],:]
    if pad[0]+pad[1] > 0:
        x = x[:,:,:,pad[0]:-pad[1]]
    return x

A test snippet:
x = torch.zeros(4, 3, 1080, 1920) # Raw data
x_pad, pads = pad_to(x, 16) # Padded data, feed this to your network
x_unpad = unpad(x_pad, pads) # Un-pad the network output to recover the original shape

print('Original: ', x.shape)
print('Padded: ', x_pad.shape)
print('Recovered: ', x_unpad.shape)

Output:
Original:  torch.Size([4, 3, 1080, 1920])
Padded:  torch.Size([4, 3, 1088, 1920])
Recovered:  torch.Size([4, 3, 1080, 1920])

Reference: https://github.com/seoungwugoh/STM/blob/905f11492a6692dd0d0fa395881a8ec09b211a36/helpers.py#L33


Instruction: How to check whether tensor values in a different tensor pytorch?

Input:
I have 2 tensors of unequal size
a = torch.tensor([[1,2], [2,3],[3,4]])
b = torch.tensor([[4,5],[2,3]])

I want a boolean array of whether each value exists in the other tensor without iterating. something like
a in b

and the result should be
[False, True, False]

as only the value of a[1] is in b

Output:
I think it's impossible without using at least some type of iteration. The most succinct way I can manage is using list comprehension:
[True if i in b else False for i in a]

Checks for elements in b that are in a and gives [False, True, False]. Can also be reversed to get elements a in b [False, True].


Instruction: How to save a model using DefaultTrainer in Detectron2?

Input:
How can I save a checkpoint in Detectron2, using a DefaultTrainer?
This is my setup:
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))

cfg.DATASETS.TRAIN = (DatasetLabels.TRAIN,)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 273  # Number of output classes

cfg.OUTPUT_DIR = "outputs"
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025#0.00025  # Learning Rate
cfg.SOLVER.MAX_ITER = 10000  # 20000 MAx Iterations
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128  # Batch Size

trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

# Save the model
from detectron2.checkpoint import DetectionCheckpointer, Checkpointer
checkpointer = DetectionCheckpointer(trainer, save_dir=cfg.OUTPUT_DIR)
checkpointer.save("mymodel_0")

I get the error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-94-c1116902655a> in <module>()

      4 checkpointer = DetectionCheckpointer(trainer, save_dir=cfg.OUTPUT_DIR)
----> 5 checkpointer.save("mymodel_0")

/usr/local/lib/python3.6/dist-packages/fvcore/common/checkpoint.py in save(self, name, **kwargs)
    102
    103         data = {}
--> 104         data["model"] = self.model.state_dict()
    105         for key, obj in self.checkpointables.items():
    106             data[key] = obj.state_dict()

AttributeError: 'DefaultTrainer' object has no attribute 'state_dict'

Docs: https://detectron2.readthedocs.io/en/latest/modules/checkpoint.html

Output:
checkpointer = DetectionCheckpointer(trainer.model, save_dir=cfg.OUTPUT_DIR)

is the way to go.
Alternatively:
torch.save(trainer.model.state_dict(), os.path.join(cfg.OUTPUT_DIR, "mymodel.pth"))


Instruction: PyTorch: Predicting future values with LSTM

Input:
I'm currently working on building an LSTM model to forecast time-series data using PyTorch. I used lag features to pass the previous n steps as inputs to train the network. I split the data into three sets, i.e., train-validation-test split, and used the first two to train the model. My validation function takes the data from the validation data set and calculates the predicted valued by passing it to the LSTM model using DataLoaders and TensorDataset classes. Initially, I've got pretty good results with R2 values in the region of 0.85-0.95.
However, I have an uneasy feeling about whether this validation function is also suitable for testing my model's performance. Because the function now takes the actual X values, i.e., time-lag features, from the DataLoader to predict y^ values, i.e., predicted target values, instead of using the predicted y^ values as features in the next prediction. This situation seems far from reality where the model has no clue of the real values of the previous time steps, especially if you forecast time-series data for longer time periods, say 3-6 months.
I'm currently a bit puzzled about tackling this issue and defining a function to predict future values relying on the model's values rather than the actual values in the test set. I have the following function predict, which makes a one-step prediction, but I haven't really figured out how to predict the whole test dataset using DataLoader.
    def predict(self, x):
        # convert row to data
        x = x.to(device)
        # make prediction
        yhat = self.model(x)
        # retrieve numpy array
        yhat = yhat.to(device).detach().numpy()
        return yhat

You can find how I split and load my datasets, my constructor for the LSTM model, and the validation function below. If you need more information, please do not hesitate to reach out to me.
Splitting and Loading Datasets
def create_tensor_datasets(X_train_arr, X_val_arr, X_test_arr, y_train_arr, y_val_arr, y_test_arr):
    train_features = torch.Tensor(X_train_arr)
    train_targets = torch.Tensor(y_train_arr)
    val_features = torch.Tensor(X_val_arr)
    val_targets = torch.Tensor(y_val_arr)
    test_features = torch.Tensor(X_test_arr)
    test_targets = torch.Tensor(y_test_arr)

    train = TensorDataset(train_features, train_targets)
    val = TensorDataset(val_features, val_targets)
    test = TensorDataset(test_features, test_targets)

    return train, val, test

def load_tensor_datasets(train, val, test, batch_size=64, shuffle=False, drop_last=True):
    train_loader = DataLoader(train, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last)
    val_loader = DataLoader(val, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last)
    test_loader = DataLoader(test, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last)
    return train_loader, val_loader, test_loader

Class LSTM
class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, dropout_prob):
        super(LSTMModel, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm = nn.LSTM(
            input_dim, hidden_dim, layer_dim, batch_first=True, dropout=dropout_prob
        )

        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x, future=False):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = out[:, -1, :]
        out = self.fc(out)

        return out

Validation (defined within a trainer class)
    def validation(self, val_loader, batch_size, n_features):

        with torch.no_grad():
            predictions = []
            values = []
            for x_val, y_val in val_loader:
                x_val = x_val.view([batch_size, -1, n_features]).to(device)
                y_val = y_val.to(device)
                self.model.eval()
                yhat = self.model(x_val)
                predictions.append(yhat.cpu().detach().numpy())
                values.append(y_val.cpu().detach().numpy())

        return predictions, values

Output:
I've finally found a way to forecast values based on predicted values from the earlier observations. As expected, the predictions were rather accurate in the short-term, slightly becoming worse in the long term. It is not so surprising that the future predictions digress over time, as they no longer depend on the actual values. Reflecting on my results and the discussions I had on the topic, here are my take-aways:

In real-life cases, the real values can be retrieved and fed into the model at each step of the prediction -be it weekly, daily, or hourly- so that the next step can be predicted with the actual values from the previous step. So, testing the performance based on the actual values from the test set may somewhat reflect the real performance of the model that is maintained regularly.

However, for predicting future values in the long term, forecasting, if you will, you need to make either multiple one-step predictions or multi-step predictions that span over the time period you wish to forecast.

Making multiple one-step predictions based on the values predicted the model yields plausible results in the short term. As the forecasting period increases, the predictions become less accurate and therefore less fit for the purpose of forecasting.

To make multiple one-step predictions and update the input after each prediction, we have to work our way through the dataset one by one, as if we are going through a for-loop over the test set. Not surprisingly, this makes us lose all the computational advantages that matrix operations and mini-batch training provide us.

An alternative could be predicting sequences of values, instead of predicting the next value only, say using RNNs with multi-dimensional output with many-to-many or seq-to-seq structure. They are likely to be more difficult to train and less flexible to make predictions for different time periods. An encoder-decoder structure may prove useful for solving this, though I have not implemented it by myself.

You can find the code for my function that forecasts the next n_steps based on the last row of the dataset X (time-lag features) and y (target value). To iterate over each row in my dataset, I would set batch_size to 1 and n_features to the number of lagged observations.
    def forecast(self, X, y, batch_size=1, n_features=1, n_steps=100):
        predictions = []
        X = torch.roll(X, shifts=1, dims=2)
        X[..., -1, 0] = y.item(0)
        with torch.no_grad():
            self.model.eval()
            for _ in range(n_steps):
                X = X.view([batch_size, -1, n_features]).to(device)
                yhat = self.model(X)
                yhat = yhat.to(device).detach().numpy()
                X = torch.roll(X, shifts=1, dims=2)
                X[..., -1, 0] = yhat.item(0)
                predictions.append(yhat)

        return predictions

The following line shifts values in the second dimension of the tensor by one so that a tensor [[[x1, x2, x3, ... , xn ]]] becomes [[[xn, x1, x2, ... , x(n-1)]]].
X = torch.roll(X, shifts=1, dims=2)
And, the line below selects the first element from the last dimension of the 3d tensor and sets that item to the predicted value stored in the NumPy ndarray (yhat), [[xn+1]]. Then, the new input tensor becomes [[[x(n+1), x1, x2, ... , x(n-1)]]]
X[..., -1, 0] = yhat.item(0)
Recently, I've decided to put together the things I had learned and the things I would have liked to know earlier. If you'd like to have a look, you can find the links down below. I hope you'll find it useful. Feel free to comment or reach out to me if you agree or disagree with any of the remarks I made above.

Building RNN, LSTM, and GRU for time series using PyTorch
Predicting future values with RNN, LSTM, and GRU using PyTorch


Instruction: Binary classification with PyTorch

Input:
Below is code I've written for binary classification in PyTorch:
%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F

device = 'cpu'
num_epochs = 10

hidden_size = 500
num_classes = 2

learning_rate = .001

torch.manual_seed(24)

x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])
x = torch.tensor([x1,x2,x3,x4]).float()
y = torch.tensor([1,0,1,1]).long()

train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)

input_size = len(x[0])

def weights_init(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0.0, 1)

class NeuralNet(nn.Module) :
    def __init__(self, input_size, hidden_size, num_classes) :
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size , hidden_size)
        self.fc2 = nn.Linear(hidden_size , 100)
        self.fc3 = nn.Linear(100 , num_classes)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x) :
        out = self.fc1(x)
        out = self.sigmoid(out)
        out = self.fc2(out)
        out = self.sigmoid(out)
        out = self.fc3(out)
        out = self.sigmoid(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)
model.apply(weights_init)

criterionCE = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
loss_values = []

for i in range(0 , 140) :
    print('in i' , i)
    total_step = len(train_loader)
    for epoch in range(num_epochs) :
        for i,(images , labels) in enumerate(train_loader) :
            images = images.to(device)
            labels = labels.to(device)

            outputs = model(images)
            print('outputs' , outputs)
            loss = criterionCE(outputs , labels)
            loss_values.append(loss)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

    outputs = model(x)

    print(outputs.data.max(1)[1])

This code throws an error :
~/opt/miniconda3/envs/ds1/lib/python3.8/site-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2067                       stacklevel=2)
   2068     if input.numel() != target.numel():
-> 2069         raise ValueError("Target and input must have the same number of elements. target nelement ({}) "
   2070                          "!= input nelement ({})".format(target.numel(), input.numel()))
   2071

ValueError: Target and input must have the same number of elements. target nelement (2) != input nelement (4)

I printed where I think is the cause of the error:
print('outputs' , outputs)

renders
outputs tensor([[9.9988e-01, 1.4011e-05],
        [9.9325e-01, 1.2087e-05]], grad_fn=<SigmoidBackward>)

So outputs should be a 2x1 instead of a 2x2 result. Have I not defined the setup of the model correctly? As I'm using sigmoid shouldn't a 2x1 output be created instead of 2x2?

Output: If you are working on a binary classification task your model should only output one logit. Since you've set self.fc3 to have 2 neurons, you will get 2 logits as the output. Therefore, you should set self.fc3 as nn.Linear(100 , 1).


Instruction: Why is the timm visual transformer position embedding initializing to zeros?

Input:
I'm looking at the timm implementation of visual transformers and for the positional embedding, he is initializing his position embedding with zeros as follows:
self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, embed_dim))

See here:
https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py#L309
I'm not sure how this actually embeds anything about the position when it is later added to the patch?
x = x + self.pos_embed

Any feedback is appreciated.

Output: The positional embedding is a parameter that gets included in the computational graph and gets updated during training. So, it doesn't matter if you initialize with zeros; they are learned during training.


Instruction: How to implement randomised log space search of learning rate in PyTorch?

Input:
I am looking to fine tune a GNN and my supervisor suggested exploring different learning rates. I came across this tutorial video where he mentions that a randomised log space search of hyper parameters is typically done in practice. For sake of the introductory tutorial this was not covered.
Any help or pointers on how to achieve this in PyTorch is greatly appreciated. Thank you!

Output:
Setting the scale in logarithm terms let you take into account more desirable values of the learning rate, usually values lower than 0.1
Imagine you want to take learning rate values between 0.1 (1e-1) and 0.001 (1e-4). Then you can set this lower and upper bound on a logarithm scale by applying a logarithm base 10 on it,  log10(0.1) = -1 and log10(0.001) = -4. Andrew Ng provides a clearer explanation in this video.
In Python you can use np.random.uniform() for this
searchable_learning_rates = [10**(-4 * np.random.uniform(0.5, 1)) for _ in range(10)]
searchable_learning_rates
>>>
[0.004890650359810075,
 0.007894672127828331,
 0.008698831627963768,
 0.00022779163472045743,
 0.0012046829055603172,
 0.00071395500159473,
 0.005690032483124896,
 0.000343368839731761,
 0.0002819402550629178,
 0.0006399571804618883]

as you can see you're able to try learning rate values from 0.0002819402550629178 up to 0.008698831627963768 which is close to the upper bound. The longer the array the more values you will try.
Following the example code in the video you provided you can implement the randomized log search for the learning rate by replacing learning_rates for searchable learning_rates
for batch_size in batch_sizes:
    for learning_rate in searchable_learning_rates:
    ...
    ...


Instruction: why torch.Tensor subtract works well when tensor size is different?

Input:
This example will make it easier to understand. The following fails:
A = tensor.torch([[1, 2, 3], [4, 5, 6]])   # shape : (2, 3)
B = tensor.torch([[1, 2], [3, 4], [5, 6]]) # shape : (3, 2)
print((A - B).shape)

# RuntimeError: The size of tensor A (3) must match the size of tensor B (2) at non-singleton dimension 1
# ==================================================================
A = tensor.torch([[1, 2], [3, 4], [5, 6]])   # shape : (3, 2)
B = tensor.torch([[1, 2], [3, 4],]) # shape : (2, 2)
print((A - B).shape)

# RuntimeError: The size of tensor A (3) must match the size of tensor B (2) at non-singleton dimension 0

But the following works well:
a = torch.ones(8).unsqueeze(0).unsqueeze(-1).expand(4, 8, 7)
a_temp = a.unsqueeze(2)                            # shape : ( 4, 8, 1, 7 )
b_temp = torch.transpose(a_temp, 1, 2)             # shape : ( 4, 1, 8, 7 )
print(a_temp-b_temp)                               # shape : ( 4, 8, 8, 7 )

Why does the latter work, but not the former?
How/why has the result shape been expanded?

Output:
This is well explained by the broadcasting semantics. The important part is :
Two tensors are “broadcastable” if the following rules hold:

Each tensor has at least one dimension.
When iterating over the dimension sizes, starting at the trailing dimension, the dimension sizes must either be equal, one of them is 1, or one of them does not exist.

In your case, (3,2) and (2,3) cannot be broadcast to a common shape (3 != 2 and neither are equal to 1), but (4,8,1,7), (4,1,8,7) and (4,8,8,7) are broadcast compatible.
This is basically what the error states : all dimensions must be either equal ("match") or singletons (i.e equal to 1)
What happens when the shape are broadcasted is basically a tensor expansion to make the shape match (expand to [4,8,8,7]), and then perform the subtraction as usual. Expansion duplicates your data (in a smart way) to reach the required shape.


Instruction: How to actually apply a Conv2d filter in Pytorch

Input:
I’m new to Python and trying to do some manipulations with filters in PyTorch.
I’m struggling re how to apply a Conv2d. I’ve got the following code which creates a 3x3 moving average filter:
resized_image4D = np.reshape(image_noisy, (1, 1, image_noisy.shape[0], image_noisy.shape[1]))
t = torch.from_numpy(resized_image4D)

conv = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, padding=1, bias=False)
conv.weight = torch.nn.Parameter(torch.ones((1,1,3, 3))/9.0)

Normally in NumPy I’d just call filtered_image = convolve2d(image, kernel), but I’ve not been able to figure out what the PyTorch equivalent is after days of searching.

Output:
I think you are looking for torch.nn.functional.conv2d.
Hence, your snippets becomes:
resized_image4D = np.reshape(image_noisy, (1, 1, image_noisy.shape[0], image_noisy.shape[1]))
t = torch.from_numpy(resized_image4D)

conv = torch.nn.functional.conv2d(in_channels=1, out_channels=1, kernel_size=3, padding=1, bias=False)
conv.weight = torch.nn.Parameter(torch.ones((1,1,3, 3))/9.0)


Instruction: How to train faster-rcnn on dataset including negative data in pytorch

Input:
I am trying to train the torchvision Faster R-CNN model for object detection on my custom data. I used the code in torchvision object detection fine-tuning tutorial. But getting this error:
Expected target boxes to be a tensor of shape [N, 4], got torch.Size([0])

This has to do with negative data (empty training images / no bounding boxes) in my custom dataset. How can we change the below Dataset class to enable training faster-rcnn on dataset including negative data?
class MyCustomDataset(Dataset):

    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "PNGImages"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "PedMasks"))))

    def __len__(self):
        return len(self.imgs)

    def __getitem__(self, idx):
        # load images ad masks
        img_path = os.path.join(self.root, "PNGImages", self.imgs[idx])
        mask_path = os.path.join(self.root, "PedMasks", self.masks[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)
        # convert the PIL Image into a numpy array
        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)

        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        # convert everything into a torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["image_id"] =  torch.tensor([idx])
        target["area"] = area
        target["iscrowd"] = iscrowd

        return img, target

Output:
We need to make two changes to the Dataset Class.
1- Empty boxes are fed as:
if num_objs == 0:
    boxes = torch.zeros((0, 4), dtype=torch.float32)
else:
    boxes = torch.as_tensor(boxes, dtype=torch.float32)

2- Assign area=0 for empty bounding box case, change the code used for calculating area, and make it a torch tensor:
area = 0
for i in range(num_objs):
    pos = np.where(masks[i])
    xmin = np.min(pos[1])
    xmax = np.max(pos[1])
    ymin = np.min(pos[0])
    ymax = np.max(pos[0])
    area += (xmax-xmin)*(ymax-ymin)
area = torch.as_tensor(area, dtype=torch.float32)

We will incorporate step#2 in existing for loop.
So, the modified Dataset Class will look like:
class MyCustomDataset(Dataset):

    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "PNGImages"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "PedMasks"))))

    def __len__(self):
        return len(self.imgs)

    def __getitem__(self, idx):
        # load images ad masks
        img_path = os.path.join(self.root, "PNGImages", self.imgs[idx])
        mask_path = os.path.join(self.root, "PedMasks", self.masks[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)
        # convert the PIL Image into a numpy array
        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)

        boxes = []
        area = 0
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])
            area += (xmax-xmin)*(ymax-ymin)
        area = torch.as_tensor(area, dtype=torch.float32)

        # Handle empty bounding boxes
        if num_objs == 0:
            boxes = torch.zeros((0, 4), dtype=torch.float32)
        else:
            boxes = torch.as_tensor(boxes, dtype=torch.float32)

        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        image_id = torch.tensor([idx])

        #area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])

        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["image_id"] =  torch.tensor([idx])
        target["area"] = area
        target["iscrowd"] = iscrowd

        return img, target


Instruction: Customizing the batch with specific elements

Input:
I am a fresh starter with PyTorch. Strangely I cannot find anything related to this, although it seems rather simple.
I want to structure my batch with specific examples, like all examples per batch having the same label, or just fill the batch with examples of just 2 classes.
How would I do that? For me, it seems the right place within the data loader and not in the dataset? As the data loader is responsible for the batches and not the dataset?
Is there a simple minimal example?

Output:
TLDR;

Default DataLoader only uses a sampler, not a batch sampler.

You can define a sampler, plus a batch sampler, a batch sampler will override the sampler.

The sampler only yields the sequence of dataset elements, not the actual batches (this is handled by the data loader, depending on batch_size).

To answer your initial question: Working with a sampler on an iterable dataset doesn't seem to be possible cf. Github issue (still open). Also, read the following note on pytorch/dataloader.py.

Samplers (for map-style datasets):
That aside, if you are switching to a map-style dataset, here are some details on how samplers and batch samplers work. You have access to a dataset's underlying data using indices, just like you would with a list (since torch.utils.data.Dataset implements __getitem__). In other words, your dataset elements are all dataset[i], for i in [0, len(dataset) - 1].
Here is a toy dataset:
class DS(Dataset):
    def __getitem__(self, index):
        return index

    def __len__(self):
        return 10

In a general use case you would just give torch.utils.data.DataLoader the arguments batch_size and shuffle. By default, shuffle is set to false, which means it will use torch.utils.data.SequentialSampler. Else (if shuffle is true) torch.utils.data.RandomSampler will be used. The sampler defines how the data loader accesses the dataset (in which order it accesses it).
The above dataset (DS) has 10 elements. The indices are 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. They map to elements 0, 10, 20, 30, 40, 50, 60, 70, 80, and 90. So with a batch size of 2:

SequentialSampler: DataLoader(ds, batch_size=2) (implictly shuffle=False), identical to DataLoader(ds, batch_size=2, sampler=SequentialSampler(ds)). The dataloader will deliver tensor([0, 10]), tensor([20, 30]), tensor([40, 50]), tensor([60, 70]), and tensor([80, 90]).

RandomSampler: DataLoader(ds, batch_size=2, shuffle=True), identical to DataLoader(ds, batch_size=2, sampler=RandomSampler(ds)). The dataloader will sample randomly each time you iterate through it. For instance: tensor([50, 40]), tensor([90, 80]), tensor([0, 60]),  tensor([10, 20]), and tensor([30, 70]). But the sequence will be different if you iterate through the dataloader a second time!

Batch sampler
Providing batch_sampler will override batch_size, shuffle, sampler, and drop_last altogether. It is meant to define exactly the batch elements and their content. For instance:
>>> DataLoader(ds, batch_sampler=[[1,2,3], [6,5,4], [7,8], [0,9]])`

Will yield tensor([10, 20, 30]), tensor([60, 50, 40]), tensor([70, 80]), and tensor([ 0, 90]).

Batch sampling on the class

Let's say I just want to have 2 elements (different or not) of each class in my batch and have to exclude more examples of each class. So ensuring that not 3 examples are inside of the batch.

Let's say you have a dataset with four classes. Here is how I would do it. First, keep track of dataset indices for each class.
class DS(Dataset):
    def __init__(self, data):
        super(DS, self).__init__()
        self.data = data

        self.indices = [[] for _ in range(4)]
        for i, x in enumerate(data):
            if x > 0 and x % 2: self.indices[0].append(i)
            if x > 0 and not x % 2: self.indices[1].append(i)
            if x < 0 and x % 2: self.indices[2].append(i)
            if x < 0 and not x % 2: self.indices[3].append(i)

    def classes(self):
        return self.indices

    def __getitem__(self, index):
        return self.data[index]

For example:
>>> ds = DS([1, 6, 7, -5, 10, -6, 8, 6, 1, -3, 9, -21, -13, 11, -2, -4, -21, 4])

Will give:
>>> ds.classes()
[[0, 2, 8, 10, 13], [1, 4, 6, 7, 17], [3, 9, 11, 12, 16], [5, 14, 15]]

Then for the batch sampler, the easiest way is to create a list of class indices that are available, and have as many class index as there are dataset element.
In the dataset defined above, we have 5 items from class 0, 5 from class 1, 5 from class 2, and 3 from class 3. Therefore we want to construct [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3]. We will shuffle it. Then, from this list and the dataset classes content (ds.classes()) we will be able to construct the batches.
class Sampler():
    def __init__(self, classes):
        self.classes = classes

    def __iter__(self):
        classes = copy.deepcopy(self.classes)
        indices = flatten([[i for _ in range(len(klass))] for i, klass in enumerate(classes)])
        random.shuffle(indices)
        grouped = zip(*[iter(indices)]*2)

        res = []
        for a, b in grouped:
            res.append((classes[a].pop(), classes[b].pop()))
        return iter(res)

Note - deep copying the list is required since we're popping elements from it.
A possible output of this sampler would be:
[(15, 14), (16, 17), (7, 12), (11, 6), (13, 10), (5, 4), (9, 8), (2, 0), (3, 1)]

At this point we can simply use torch.data.utils.DataLoader:
>>> dl = DataLoader(ds, batch_sampler=sampler(ds.classes()))

Which could yield something like:
[tensor([ 4, -4]), tensor([-21,  11]), tensor([-13,   6]), tensor([9, 1]), tensor([  8, -21]), tensor([-3, 10]), tensor([ 6, -2]), tensor([-5,  7]), tensor([-6,  1])]

An easier approach
Here is another - easier - approach that will not guarantee to return all elements from the dataset, on average it will...
For each batch, first sample class_per_batch classes, then sample batch_size elements from these selected classes (by first sampling a class from that class subset, then sampling from a data point from that class).
class Sampler():
    def __init__(self, classes, class_per_batch, batch_size):
        self.classes = classes
        self.n_batches = sum([len(x) for x in classes]) // batch_size
        self.class_per_batch = class_per_batch
        self.batch_size = batch_size

    def __iter__(self):
        classes = random.sample(range(len(self.classes)), self.class_per_batch)

        batches = []
        for _ in range(self.n_batches):
            batch = []
            for i in range(self.batch_size):
                klass = random.choice(classes)
                batch.append(random.choice(self.classes[klass]))
            batches.append(batch)
        return iter(batches)

You can try it this way:
>>> s = Sampler(ds.classes(), class_per_batch=2, batch_size=4)
>>> list(s)
[[16, 0, 0, 9], [10, 8, 11, 2], [16, 9, 16, 8], [2, 9, 2, 3]]

>>> dl = DataLoader(ds, batch_sampler=s)
>>> list(iter(dl))
[tensor([ -5,  -6, -21, -13]), tensor([ -4,  -4, -13, -13]), tensor([ -3, -21,  -2,  -5]), tensor([-3, -5, -4, -6])]


Instruction: Why doesn't this input pass through this simple PyTorch model?

Input:
I have an input/tensor whose shape is:
torch.Size([256, 3, 28, 28])

(batch size here is 256, 3 channels, 28x28 image)
And a model like so:
class Model(nn.Module):

    def __init__(self):
        super().__init__()
        self.network = nn.Sequential(
            nn.Conv2d(3, 28, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(28, 56, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),  # output: 56 x 16 x 16

            nn.Conv2d(56, 112, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(112, 112, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),  # output: 112 x 8 x 8

            nn.Conv2d(112, 224, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(224, 224, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),  # output: 224 x 4 x 4

            nn.Flatten(),
            nn.Linear(224 * 4 * 4, 896),
            nn.ReLU(),
            nn.Linear(896, 512),
            nn.ReLU(),
            nn.Linear(512, 2))

    def forward(self, xb):
        return self.network(xb)

When I attempt to pass data forward, it fails with:
    ...
    return self.network(xb)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/nn/functional.py", line 1690, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: mat1 dim 1 must match mat2 dim 0

What am I missing?
Thanks!

Output:
nn.MaxPool2d(2, 2),  # output: 56 x 16 x 16

This is wrong. The original input is of size (256, 3, 28, 28). The convolutional and ReLU layers you're using do not change the batch, height, or width dimensions; they only change the "channel" dimension. Prior to the max pooling layer, the tensor size is (256, 56, 28, 28). The max pooling layer has a kernel size of 2 and a stride of 2, so it will cut both the height and width in half. So the output of this max pooling layer is of size (256, 56, 14, 14).
For the same reason, the output of the next max pooling layer is of size (256, 112, 7, 7), and the output of the last max pooling layer is of size (256, 224, 3, 3).
So you can either fix this by changing your input size to (256, 3, 32, 32), if that's an option, or changing the first linear layer to nn.Linear(224 * 3 * 3, 896)


Instruction: BERT Convert 'SpanAnnotation' to answers using scores from hugging face models

Input:
I'm following along with the documentation for importing a pretrained model question and answer model from huggingface
from transformers import BertTokenizer, BertForQuestionAnswering
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-base-uncased')
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
inputs = tokenizer(question, text, return_tensors='pt')
start_positions = torch.tensor([1])
end_positions = torch.tensor([3])
outputs = model(**inputs, start_positions=start_positions, end_positions=end_positions)
loss = outputs.loss
start_scores = outputs.start_logits
end_scores = outputs.end_logits

this returns start and end scores, but how can I get a meaningful text answer from here?

Output:
So I did a little digging around and it looks like scores can be converted to tokens which can be used to build the answer.  Here is a short example:
answer_start = torch.argmax(start_scores)
answer_end = torch.argmax(end_scores) + 1

tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(inputs["input_ids"][0][answer_start:answer_end]))


Instruction: "RuntimeError: expected scalar type Double but found Float" in Pytorch CNN training

Input:
I just begin to learn Pytorch and create my first CNN. The dataset contains 3360 RGB images and I converted them to a [3360, 3, 224, 224] tensor. The data and label are in the dataset(torch.utils.data.TensorDataset). Below is the training code.
    def train_net():
        dataset = ld.load()
        data_iter = Data.DataLoader(dataset, batch_size=168, shuffle=True)
        net = model.VGG_19()
        summary(net, (3, 224, 224), device="cpu")
        loss_func = nn.CrossEntropyLoss()
        optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9, dampening=0.1)
        scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.1)
        for epoch in range(5):
            print("epoch:", epoch + 1)
            train_loss = 0
            for i, data in enumerate(data_iter, 0):
                x, y = data
                print(x.dtype)
                optimizer.zero_grad()
                out = net(x)
                loss = loss_func(out, y)
                loss.backward()
                optimizer.step()
                train_loss += loss.item()
                if i % 100 == 99:
                    print("loss:", train_loss / 100)
                    train_loss = 0.0
        print("finish train")

Then I have this error:
    Traceback (most recent call last):
              File "D:/python/DeepLearning/VGG/train.py", line 52, in <module>
                train_net()
              File "D:/python/DeepLearning/VGG/train.py", line 29, in train_net
                out = net(x)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\DeepLearning\VGG\model.py", line 37, in forward
                out = self.conv3_64(x)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\lib\site-packages\torch\nn\modules\container.py", line 117, in forward
                input = module(input)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
                return self._conv_forward(input, self.weight)
              File "D:\python\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
                return F.conv2d(input, weight, self.bias, self.stride,
            RuntimeError: expected scalar type Double but found Float

I think there is something wrong with x and I print its type by print(x.dtype):
torch.float64

which is double instead of float. Do you know what`s wrong? Thanks for your help!

Output:
that error is actually refering to the weights of the conv layer which are in float32 by default when the matrix multiplication is called. Since your input is double(float64 in pytorch)  while the weights in conv are float
So the solution in your case is :
def train_net():
    dataset = ld.load()
    data_iter = Data.DataLoader(dataset, batch_size=168, shuffle=True)
    net = model.VGG_19()
    summary(net, (3, 224, 224), device="cpu")
    loss_func = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9, dampening=0.1)
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.1)
    for epoch in range(5):
        print("epoch:", epoch + 1)
        train_loss = 0
        for i, data in enumerate(data_iter, 0):
            x, y = data                        #      //_______________
            x = x.float()    # HERE IS THE CHANGE     \\
            print(x.dtype)
            optimizer.zero_grad()
            out = net(x)
            loss = loss_func(out, y)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
            if i % 100 == 99:
                print("loss:", train_loss / 100)
                train_loss = 0.0
    print("finish train")

This will work for sure


Instruction: Pytorch nn.Linear RuntimeError: mat1 dim 1 must match mat2 dim 0

Input:
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, 3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        self.max_pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(7*7*32, 128)
        self.fc1 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.max_pool(F.relu(self.conv1(x)))
        x = self.max_pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.softmax(x, dim=1)
        return x

The above is the model. The shape of input image is 1×28×28 with batch_size 16. I tried to print x.shape after flatten, which is (16, 1568), completely corresponding to the input size of self.fc1(). Does anyone have any idea?

Output:
you have named this function nn.Linear(128, 10) as self.fc1
i.e self.fc1 = nn.Linear(128, 10)
Please change it to self.fc2 = nn.Linear(128, 10)


Instruction: Best input size of a the first layer 1D CNN

Input:
I'm trying to build a 1D CNN with time series. The input is of length 500. There are (only) 2 labels. The architecture which I built so far is the following: there are 3 convolution layers each, of them followed by an activation layer. The first convolution layer takes 50 channels as input.

import torch
import torch.nn as nn
import numpy as np
import random

class Simple1DCNN3(torch.nn.Module):
    def __init__(self):
        super(Simple1DCNN5, self).__init__()
        self.sequence = nn.Sequential(
            torch.nn.Conv1d(in_channels=50,
                                          out_channels=64,
                                          kernel_size=5,
                                          stride=2),
            torch.nn.ReLU(),
            torch.nn.Conv1d(in_channels=64,
                                          out_channels=128,
                                          kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.Conv1d(in_channels=128,
                                          out_channels=256,
                                          kernel_size=1),
            torch.nn.ReLU(),
        )
        self.fc1 = nn.Linear(256, 2)

    def forward(self, x):
        x = x.view(1, 50,-1)

        for layer in self.sequence:
            x = layer(x)
            print(x.size())
        x = x.view(1,-1)
        #print(x.size())
        x = self.fc1(x)
        #print(x.size())
        return x

net = Simple1DCNN3()

input_try = np.random.uniform(-10, 10, 500)
input_try = torch.from_numpy(input_try).float()
net(input_try)
print("input successfull passed to net")
input_try_modif = input_try.view(1, 50,-1)
print(input_try.shape)
print(input_try_modif.shape)

As far as I understood, that forced me to segment the input in 10 segments of 50 timepoints. Am I understanding it wrong ? Wouldn't it be wiser to construct the first layer with 500 channels as inputs and have a sliding window kernel? I tried it in the following other script but got the following error message

import torch
import torch.nn as nn
import numpy as np
import random

class Simple1DCNN4(torch.nn.Module):
    def __init__(self):
        super(Simple1DCNN5, self).__init__()
        self.sequence = nn.Sequential(
            torch.nn.Conv1d(in_channels=500,
                                          out_channels=64,
                                          kernel_size=5,
                                          stride=2),
            torch.nn.ReLU(),
            torch.nn.Conv1d(in_channels=64,
                                          out_channels=128,
                                          kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.Conv1d(in_channels=128,
                                          out_channels=256,
                                          kernel_size=1),
            torch.nn.ReLU(),
        )
        self.fc1 = nn.Linear(256, 2)

    def forward(self, x):
        x = x.view(1, 50,-1)

        for layer in self.sequence:
            x = layer(x)
            print(x.size())
        x = x.view(1,-1)
        #print(x.size())
        x = self.fc1(x)
        #print(x.size())
        return x

net = Simple1DCNN4()

input_try = np.random.uniform(-10, 10, 500)
input_try = torch.from_numpy(input_try).float()
net(input_try)
print("input successfull passed to net")
input_try_modif = input_try.view(1, 50,-1)
print(input_try.shape)
print(input_try_modif.shape)

Error message:
RuntimeError: Given groups=1, weight of size [64, 500, 5], expected input[1, 50, 10] to have 500 channels, but got 50 channels instead

EDIT
Thanks to the answer of @ghchoi, here is the code of the working kernel. For this, I also had to change the kernel size to all the convolutional layers to 1.
class Simple1DCNN5(torch.nn.Module):
    def __init__(self):
        super(Simple1DCNN5, self).__init__()
        self.sequence = nn.Sequential(
            torch.nn.Conv1d(in_channels=500,
                                          out_channels=64,
                                          kernel_size=1,
                                          stride=2),
            torch.nn.ReLU(),
            torch.nn.Conv1d(in_channels=64,
                                          out_channels=128,
                                          kernel_size=1),
            torch.nn.ReLU(),
            torch.nn.Conv1d(in_channels=128,
                                          out_channels=256,
                                          kernel_size=1),
            torch.nn.ReLU(),
        )
        self.fc1 = nn.Linear(256, 2)

    def forward(self, x):
        x = x.view(1, 500,-1)

        for layer in self.sequence:
            x = layer(x)
            #print(x.size())
        x = x.view(1,-1)
        #print(x.size())
        x = self.fc1(x)
        #print(x.size())
        return x

The kind of data I have is mono-derivation ECG (electrocardiogram) signal of 2 seconds. This is a recording of the electrical signal of the hear. Here is the idea of what a sample could look like (ploted on a 2D graphs), where you have time on the x-axis and voltage/amplitude on the y-axis

Output:
Try
    def forward(self, x):
        x = x.view(1, 500, -1)

        ...

net = Simple1DCNN4()

input_try = np.random.uniform(-10, 10, 5000)

In this way, the input for the first Conv1d will have the 500 channels.


Instruction: Get some layers in a pytorch model that is not defined by nn.Sequential

Input:
I have a network defined below.
class model_dnn_2(nn.Module):
    def __init__(self):
        super(model_dnn_2, self).__init__()
        self.flatten = Flatten()
        self.fc1 = nn.Linear(784, 200)
        self.fc2 = nn.Linear(200, 100)
        self.fc3 = nn.Linear(100, 100)
        self.fc4 = nn.Linear(100, 10)

    def forward(self, x):
        x = self.flatten(x)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        x = F.relu(x)
        x = self.fc4(x)

I would like to take the last two layers along with the relu functions. Using children method I get the following
>>> new_model = nn.Sequential(*list(model.children())[-2:])
>>> new_model
Sequential(
  (0): Linear(in_features=100, out_features=100, bias=True)
  (1): Linear(in_features=100, out_features=10, bias=True)
)

But I would like to have the Relu function present in between the layers-just like the original model, i.e the new model should be like:
>>> new_model
Sequential(
  (0): Linear(in_features=100, out_features=100, bias=True)
  (1): Relu()
  (2): Linear(in_features=100, out_features=10, bias=True)
)

I think the children method of the model is using the class initialization to create the model and thus the problem arises.
How can I obtain the model?

Output:
The way you implemented your model, the ReLU activations are not layers, but rather functions. When listing sub-layers (aka "children") of your module you do not see the ReLUs.
You can change your implementation:
class model_dnn_2(nn.Module):
    def __init__(self):
        super(model_dnn_2, self).__init__()
        self.layers = nn.Sequential(
          nn.Flatten(),
          nn.Linear(784, 200),
          nn.ReLU(),  # now you are using a ReLU _layer_
          nn.Linear(200, 100),
          nn.ReLU(),  # this is a different ReLU _layer_
          nn.Linear(100, 100),
          nn.ReLU(),
          nn.Linear(100, 10)
        )

    def forward(self, x):
      y = self.layers(x)
      return y

More on the difference between layers and functions can be found here.


Instruction: Calculate padding for 3D CNN in Pytorch

Input:
I'm currently trying to apply a 3D CNN to a set of images with the dimensions of 193 x 229 x 193 and would like to retain the same image dimensions through each convolutional layer (similar to tensorflow's padding=SAME). I know that the padding can be calculated as follow:
S=Stride
P=Padding
W=Width
K=Kernal size

P = ((S-1)*W-S+K)/2

Which yields a padding of 1 for the first layer:
P = ((1-1)*193-1+3)/2
P= 1.0

Although I also get a result of 1.0 for each of the subsequent layers. Anyone have any suggestions? Sorry, beginner here!
Reproducible example:
import torch
import torch.nn as nn

x = torch.randn(1, 1, 193, 229, 193)

padding = ((1-1)*96-1+3)/2
print(padding)

x = nn.Conv3d(in_channels=1, out_channels=8, kernel_size=3, padding=1)(x)
print("shape after conv1: " + str(x.shape))
x = nn.Conv3d(in_channels=8, out_channels=8, kernel_size=3,padding=1)(x)
x = nn.BatchNorm3d(8)(x)
print("shape after conv2 + batch norm: " + str(x.shape))
x = nn.ReLU()(x)
print("shape after reLU:" + str(x.shape))
x = nn.MaxPool3d(kernel_size=2, stride=2)(x)
print("shape after max pool" + str(x.shape))
x = nn.Conv3d(in_channels=8, out_channels=16, kernel_size=3,padding=1)(x)
print("shape after conv3: " + str(x.shape))
x = nn.Conv3d(in_channels=16, out_channels=16, kernel_size=3,padding=1)(x)
print("shape after conv4: " + str(x.shape))

Current output:
shape after conv1: torch.Size([1, 8, 193, 229, 193])
shape after conv2 + batch norm: torch.Size([1, 8, 193, 229, 193])
shape after reLU:torch.Size([1, 8, 193, 229, 193])
shape after max pooltorch.Size([1, 8, 96, 114, 96])
shape after conv3: torch.Size([1, 16, 96, 114, 96])
shape after conv4: torch.Size([1, 16, 96, 114, 96])

Desired output:
shape after conv1: torch.Size([1, 8, 193, 229, 193])
shape after conv2 + batch norm: torch.Size([1, 8, 193, 229, 193])
...
shape after conv3: torch.Size([1, 16, 193, 229, 193])
shape after conv4: torch.Size([1, 16, 193, 229, 193])

Output:
TLDR; your formula also applies to nn.MaxPool3d
You are using a max pool layer of kernel size 2 (implicitly (2,2,2)) with a stride of 2 (implicitly (2,2,2)). This means for every 2x2x2 block you're only getting a single value. In other words - as the name implies: only the maximum value from every 2x2x2 block is pooled to the output array.
That's why you're going from (1, 8, 193, 229, 193) to (1, 8, 96, 114, 96) (notice the division by 2).
Of course, if you set kernel_size=3 and stride=1 on nn.MaxPool3d, you will preserve the shape of your blocks.

Let #x be the input shape, and #w the kernel shape. If we want the output to have the same size, then #x = floor((#x + 2p - #w)/s + 1) needs to be true. That's 2p = s(#x - 1) - #x + #w = #x(s - 1) + #w - s (your formula)
Since s = 2 and #w = 2, then 2p = #x which is not possible.


Instruction: Back-Propagation of y = x / sum(x, dim=0) where size of tensor x is (H,W)

Input:
Q1.
I'm trying to make my custom autograd function with pytorch.
But I had a problem with making analytical back propagation with  y = x / sum(x, dim=0)
where size of tensor x is (Height, Width) (x is 2-dimensional).
Here's my code
class MyFunc(torch.autograd.Function):
@staticmethod
def forward(ctx, input):
  ctx.save_for_backward(input)
  input = input / torch.sum(input, dim=0)

  return input

@staticmethod
def backward(ctx, grad_output):
  input = ctx.saved_tensors[0]
  H, W = input.size()
  sum = torch.sum(input, dim=0)
  grad_input = grad_output * (1/sum - input*1/sum**2)

  return grad_input

I used (torch.autograd import) gradcheck to compare Jacobian matrix,
from torch.autograd import gradcheck
func = MyFunc.apply
input = (torch.randn(3,3,dtype=torch.double,requires_grad=True))
test = gradcheck(func, input)

and the result was

Please someone help me to get correct back propagation result
Thanks!

Q2.
Thanks for answers!
Because of your help, I could implement back propagation in case of (H,W) tensor.
However, while I implemented back propagation in case of (N,H,W) tensor, I got a problem.
I think the problem would be initializing new tensor.
Here's my new code
import torch
import torch.nn as nn
import torch.nn.functional as F

class MyFunc(torch.autograd.Function):
  @staticmethod
  def forward(ctx, input):
    ctx.save_for_backward(input)

    N = input.size(0)
    for n in range(N):
      input[n] /= torch.sum(input[n], dim=0)

    return input

  @staticmethod
  def backward(ctx, grad_output):
    input = ctx.saved_tensors[0]
    N, H, W = input.size()
    I = torch.eye(H).unsqueeze(-1)
    sum = input.sum(1)

    grad_input = torch.zeros((N,H,W), dtype = torch.double, requires_grad=True)
    for n in range(N):
      grad_input[n] = ((sum[n] * I - input[n]) * grad_output[n] / sum[n]**2).sum(1)

    return grad_input

Gradcheck code is
from torch.autograd import gradcheck
func = MyFunc.apply
input = (torch.rand(2,2,2,dtype=torch.double,requires_grad=True))
test = gradcheck(func, input)
print(test)

and result is
enter image description here
I don't know why the error occurs...
Your help will be very helpful for me to implement my own convolutional network.
Thanks! Have a nice day.

Output:
Let's look an example with a single column, for instance: [[x1], [x2], [x3]].
Let sum be x1 + x2 + x3, then normalizing x will give y = [[y1], [y2], [y3]] = [[x1/sum], [x2/sum], [x3/sum]]. You're looking for dL/dx1, dL/x2, and dL/x3 - we'll just write them as: dx1, dx2, and dx3. Same for all dL/dyi.
So dx1 is equal to dL/dy1*dy1/dx1 + dL/dy2*dy2/dx1 + dL/dy3*dy3/dx1. That's because x1 contributes to all ouput element on the corresponding column: y1, y2, and y3.
We have:

dy1/dx1 = d(x1/sum)/dx1 = (sum - x1)/sum²

dy2/dx1 = d(x2/sum)/dx1 = -x2/sum²

similarly, dy3/dx1 = d(x3/sum)/dx1 = -x3/sum²

Therefore dx1 = (sum - x1)/sum²*dy1 - x2/sum²*dy2 - x3/sum²*dy3. Same for dx2 and dx3. As a result, the Jacobian is [dxi]_i = (sum - xi)/sum² and [dxi]_j = -xj/sum² (for all j different to i).
In your implementation, you seem to be missing all non-diagonal components.
Keeping the same one-column example, with x1=2, x2=3, and x3=5:
>>> x = torch.tensor([[2.], [3.], [5.]])

>>> sum = input.sum(0)
tensor([10])

The Jacobian will be:
>>> J = (sum*torch.eye(input.size(0)) - input)/sum**2
tensor([[ 0.0800, -0.0200, -0.0200],
        [-0.0300,  0.0700, -0.0300],
        [-0.0500, -0.0500,  0.0500]])

For an implementation with multiple columns, it's a bit trickier, more specifically for the shape of the diagonal matrix. It's easier to keep the column axis last so we don't have to bother with broadcastings:
>>> x = torch.tensor([[2., 1], [3., 3], [5., 5]])
>>> sum = x.sum(0)
tensor([10.,  9.])

>>> diag = sum*torch.eye(3).unsqueeze(-1).repeat(1, 1, len(sum))
tensor([[[10.,  9.],
         [ 0.,  0.],
         [ 0.,  0.]],

        [[ 0.,  0.],
         [10.,  9.],
         [ 0.,  0.]],

        [[ 0.,  0.],
         [ 0.,  0.],
         [10.,  9.]]])

Above diag has a shape of (3, 3, 2) where the two columns are on the last axis. Notice how we didn't need to broadcast sum.
What I wouldn't have done is: torch.eye(3).unsqueeze(0).repeat(len(sum), 1, 1). Since with this kind of shape - (2, 3, 3) - you will have to use sum[:, None, None], and will need further broadcasting down the road...
The Jacobian is simply:
>>> J = (diag - x)/sum**2
tensor([[[ 0.0800,  0.0988],
         [-0.0300, -0.0370],
         [-0.0500, -0.0617]],

        [[-0.0200, -0.0123],
         [ 0.0700,  0.0741],
         [-0.0500, -0.0617]],

        [[-0.0200, -0.0123],
         [-0.0300, -0.0370],
         [ 0.0500,  0.0494]]])

You can check the results by backpropagating through the operation using an arbitrary dy vector (not with torch.ones though, you'll get 0s because of J!). After backpropagating, x.grad should equal to torch.einsum('abc,bc->ac', J, dy).


Instruction: AllenNLP DatasetReader.read returns generator instead of AllennlpDataset

Input:
While studying AllenNLP framework (version 2.0.1), I tried to implement the example code from https://guide.allennlp.org/training-and-prediction#1.
While reading the data from a Parquet file I got:
TypeError: unsupported operand type(s) for +: 'generator' and 'generator'

for the next line:
vocab = build_vocab(train_data + dev_data)

I suspect the return value should be AllennlpDataset but maybe I got it mixed up.
What did I do wrong?
Full code:
train_path = <some_path>
test_path = <some_other_path>

class ClassificationJobReader(DatasetReader):
    def __init__(self,
                 lazy: bool = False,
                 tokenizer: Tokenizer = None,
                 token_indexers: Dict[str, TokenIndexer] = None,
                 max_tokens: int = None):
        super().__init__(lazy)
        self.tokenizer = tokenizer or WhitespaceTokenizer()
        self.token_indexers = token_indexers or {'tokens': SingleIdTokenIndexer()}
        self.max_tokens = max_tokens

    def _read(self, file_path: str) -> Iterable[Instance]:
      df = pd.read_parquet(data_path)
      for idx in df.index:
        text = row['title'][idx] + ' ' + row['description'][idx]
        print(f'text : {text}')
        label = row['class_id'][idx]
        print(f'label : {label}')
        tokens = self.tokenizer.tokenize(text)
        if self.max_tokens:
            tokens = tokens[:self.max_tokens]
        text_field = TextField(tokens, self.token_indexers)
        label_field = LabelField(label)
        fields = {'text': text_field, 'label': label_field}
        yield Instance(fields)

def build_dataset_reader() -> DatasetReader:
    return ClassificationJobReader()

def read_data(reader: DatasetReader) -> Tuple[Iterable[Instance], Iterable[Instance]]:
    print("Reading data")
    training_data = reader.read(train_path)
    validation_data = reader.read(test_path)
    return training_data, validation_data

def build_vocab(instances: Iterable[Instance]) -> Vocabulary:
    print("Building the vocabulary")
    return Vocabulary.from_instances(instances)

dataset_reader = build_dataset_reader()
train_data, dev_data = read_data(dataset_reader)
vocab = build_vocab(train_data + dev_data)

Thanks for your help

Output:
Please find below the code fix first and the explanation afterwards.

Code Fix
# the extend_from_instances expands your vocabulary with the instances passed as an arg
# and is therefore equivalent to Vocabulary.from_instances(train_data + dev_data)
# previously
vocabulary.extend_from_instances(train_data)
vocabulary.extend_from_instances(dev_data)

Explanation
This is because the AllenNLP API have had couple of breaking changes in allennlp==2.0.1. You can find the changelog here and the upgrade guide here.The guide is outdated as per my understanding (it reflects allennlp<=1.4).
The DatasetReader returns a generator now as opposed to a List previously. DatasetReader used to have a parameter called "lazy" which was for lazy loading data. It was False by default and therefore dataset_reader.read would return a List previously. However, as of v2.0 (if i remember exactly), lazy loading is applied by default and it therefore returns a generator by default. As you know, the "+" operator has not been overridden for generator objects and therefore you cannot simply add two generators.
So, you can simply use vocab.extend_from_instances to achieve same behavior as before. Hope this helped you. If you need a full code snippet, please leave a comment below, I could post a rekated gist and share it with you.
Good day!


Instruction: how to access value of a learned parameter of an activation function in a sequential network

Input:
I am implementing my custom activation function with learnable parameters. For example this can be similar to PReLu https://pytorch.org/docs/stable/generated/torch.nn.PReLU.html which a learnable parameter a.
How can I access/view the a parameter value after training ?
Solution the issue was related to the fact that it was a sequential network and this helped solve the issue https://discuss.pytorch.org/t/access-weights-of-a-specific-module-in-nn-sequential/3627

Output:
The nn.PReLU layer is a nn.Module, like most other layers you can access the weights directly using the weight property.
>>> act = nn.PReLU()
>>> act.weight
Parameter containing:
tensor([0.2500], requires_grad=True)


Instruction: Using Pytorch how to define a tensor with indices and corresponding values

Input:
Problem
I have a list of indices and a list of values like so:
i = torch.tensor([[2, 2, 1], [2, 0, 2]])
v = torch.tensor([1, 2, 3])

I want to define a (3x3 for the example) matrix which contains the values v at the indices i (1 at position (2,2), 2 at position (2, 0) and 3 at position (1,2)):
tensor([[0, 0, 0],
        [0, 0, 3],
        [2, 0, 1]])

What I have tried
I can do it using a trick, with torch.sparse and .to_dense() but I feel that it's not the "pytorchic" way to do that nor the most efficient:
f = torch.sparse.FloatTensor(indices, values, torch.Size([3, 3]))
print(f.to_dense())

Any idea for a better solution ?
Ideally I would appreciate a solution at least as fast than the one provided above.
Of course this was just an example, no particular structure in tensors i and v are assumed (neither for the dimension).

Output:
There is an alternative, as below:
import torch

i = torch.tensor([[2, 2, 1], [2, 0, 2]])
v = torch.tensor([1, 2, 3], dtype=torch.float)   # enforcing same data-type

target = torch.zeros([3,3], dtype=torch.float)   # enforcing same data-type
target.index_put_(tuple([k for k in i]), v)

print(target)

The target tensor will be as follows:
tensor([[0., 0., 0.],
        [0., 0., 3.],
        [2., 0., 1.]])

This medium.com blog article provides a comprehensive list of all index functions for PyTorch Tensors.