Instruction: Proper way to log things when using Pytorch Lightning DDP

Input:
I was wondering what is the proper way of logging metrics when using DDP. I noticed that if I want to print something inside validation_epoch_end it will be printed twice when using 2 GPUs. I was expecting validation_epoch_end to be called only on rank 0 and to receive the outputs from all GPUs, but I am not sure this is correct anymore. Therefore I have several questions:

validation_epoch_end(self, outputs) - When using DDP does every subprocess receive the data processed from the current GPU or data processed from all GPUs, i.e. does the input parameter outputs contains the outputs of the entire validation set, from all GPUs?
If outputs is GPU/process specific what is the proper way to calculate any metric on the entire validation set in validation_epoch_end when using DDP?

I understand that I can solve the printing by checking self.global_rank == 0 and printing/logging only in that case, however I am trying to get a deeper understanding of what I am printing/logging in this case.
Here is a code snippet from my use case. I would like to be able to report f1, precision and recall on the entire validation dataset and I am wondering what is the correct way of doing it when using DDP.
    def _process_epoch_outputs(self,
                               outputs: List[Dict[str, Any]]
                               ) -> Tuple[torch.Tensor, torch.Tensor]:
        """Creates and returns tensors containing all labels and predictions

        Goes over the outputs accumulated from every batch, detaches the
        necessary tensors and stacks them together.

        Args:
            outputs (List[Dict])
        """
        all_labels = []
        all_predictions = []

        for output in outputs:
            for labels in output['labels'].detach():
                all_labels.append(labels)

            for predictions in output['predictions'].detach():
                all_predictions.append(predictions)

        all_labels = torch.stack(all_labels).long().cpu()
        all_predictions = torch.stack(all_predictions).cpu()

        return all_predictions, all_labels

    def validation_epoch_end(self, outputs: List[Dict[str, Any]]) -> None:
        """Logs f1, precision and recall on the validation set."""

        if self.global_rank == 0:
            print(f'Validation Epoch: {self.current_epoch}')

        predictions, labels = self._process_epoch_outputs(outputs)
        for i, name in enumerate(self.label_columns):

            f1, prec, recall, t = metrics.get_f1_prec_recall(predictions[:, i],
                                                             labels[:, i],
                                                             threshold=None)
            self.logger.experiment.add_scalar(f'{name}_f1/Val',
                                              f1,
                                              self.current_epoch)
            self.logger.experiment.add_scalar(f'{name}_Precision/Val',
                                              prec,
                                              self.current_epoch)
            self.logger.experiment.add_scalar(f'{name}_Recall/Val',
                                              recall,
                                              self.current_epoch)

            if self.global_rank == 0:
                print((f'F1: {f1}, Precision: {prec}, '
                       f'Recall: {recall}, Threshold {t}'))

Output:
Questions

validation_epoch_end(self, outputs) - When using DDP does every
subprocess receive the data processed from the current GPU or data
processed from all GPUs, i.e. does the input parameter outputs
contains the outputs of the entire validation set, from all GPUs?

Data processed from the current GPU only, outputs are not synchronized, there is only backward synchronization (gradients are synchronized during training and distributed to replicas of models residing on each GPU).
Imagine that all of the outputs were passed from 1000 GPUs to this poor master, it could give it an OOM very easily

If outputs is GPU/process specific what is the proper way to calculate
any metric on the entire validation set in validation_epoch_end when
using DDP?

According to documentation (emphasis mine):

When validating using a accelerator that splits data from each batch
across GPUs, sometimes you might need to aggregate them on the master
GPU for processing (dp, or ddp2).

And here is accompanying code (validation_epoch_end would receive accumulated data across multiple GPUs from single step in this case, also see the comments):
# Done per-process (GPU)
def validation_step(self, batch, batch_idx):
    x, y = batch
    y_hat = self.model(x)
    loss = F.cross_entropy(y_hat, y)
    pred = ...
    return {'loss': loss, 'pred': pred}

# Gathered data from all processes (per single step)
# Allows for accumulation so the whole data at the end of epoch
# takes less memory
def validation_step_end(self, batch_parts):
    gpu_0_prediction = batch_parts.pred[0]['pred']
    gpu_1_prediction = batch_parts.pred[1]['pred']

    # do something with both outputs
    return (batch_parts[0]['loss'] + batch_parts[1]['loss']) / 2

def validation_epoch_end(self, validation_step_outputs):
   for out in validation_step_outputs:
       # do something with preds

Tips

Focus on per-device calculations and as small number of between-GPU transfers as possible

Inside validation_step (or training_step if that's what you want, this is general) calculate f1, precision, recall and whatever else on a per-batch basis
Returns those values (say, as a dict). Now you will return 3 numbers from each device instead of (batch, outputs) (which could be significantly larger)
Inside validation_step_end get those 3 values (actually (2, 3) if you have 2 GPUs) and sum/take mean of them and return 3 values
Now validation_epoch_end will get (steps, 3) values that you can use to accumulate

It would be even better if instead of operating on list of values during validation_epoch_end you could accumulate them in another 3 values (say you have a lot of validation steps, the list could grow too large), but this should be enough.
AFAIK PyTorch-Lightning doesn't do this (e.g. instead of adding to list, apply some accumulator directly), but I might be mistaken, so any correction would be great.


Instruction: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment

Input:
I am trying to install torch with CUDA support.
Here is the result of my collect_env.py script:
PyTorch version: 1.7.1+cu101
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.9 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 1080
Nvidia driver version: 460.39
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1+cu101
[pip3] torchaudio==0.7.2
[pip3] torchvision==0.8.2+cu101
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.1.243             h6bb024c_0
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py39he8ac12f_0
[conda] mkl_fft                   1.3.0            py39h54f3939_0
[conda] mkl_random                1.0.2            py39h63df603_0
[conda] numpy                     1.19.2           py39h89c1606_0
[conda] numpy-base                1.19.2           py39h2ae0177_0
[conda] torch                     1.7.1+cu101              pypi_0    pypi
[conda] torchaudio                0.7.2                    pypi_0    pypi
[conda] torchvision               0.8.2+cu101              pypi_0    pypi

Process finished with exit code 0

Here is the output of nvcc - V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Finally, here is the output of nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39       Driver Version: 460.39       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   52C    P0    46W / 180W |    624MiB /  8116MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       873      G   /usr/lib/xorg/Xorg                101MiB |
|    0   N/A  N/A      1407      G   /usr/lib/xorg/Xorg                419MiB |
|    0   N/A  N/A      2029      G   ...AAAAAAAAA= --shared-files       90MiB |
+-----------------------------------------------------------------------------+

However, when I try to run
print(torch.cuda.is_available())

I get the following error:
UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0

I have performed a reboot, and I have followed the post-installation steps as detailed in here

Output:
Had the same issue and in my case solution was very easy, however it wasn't easy to find it. I had to remove and insert nvidia_uvm module. So:
> sudo rmmod nvidia_uvm
> sudo modprobe nvidia_uvm

That's all. Just before these command collect_env.py reported "Is CUDA available: False". After: "Is CUDA available: True"


Instruction: Could not find a version that satisfies the requirement torch==1.7.0+cpu

Input:
I want to install torch==1.7.0+cpu from requirements.txt
I have an error

Could not find a version that satisfies the requirement torch==1.7.0+cpu (from -r requirements.txt (line 42)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1)
No matching distribution found for torch==1.7.0+cpu

I am using
Python 3.6.12 (default, Aug 18 2020, 02:08:22)
[GCC 5.4.0 20160609] on linux

Output:
If you have this issue while installing your modules from requirements.txt then you could simply add the following line to the top of your requirements.txt file
-f https://download.pytorch.org/whl/torch_stable.html


Instruction: Get Pytorch - tensor values as a integer in python

Input:
I have my output of my torch tensor which looks like below
(coordinate of a bounding box in object detection)
[tensor(299., device='cuda:0'), tensor(272., device='cuda:0'), tensor(327., device='cuda:0'), tensor(350., device='cuda:0')]

I wanted to extract each of the tensor value as an int in the form of minx,miny,maxx,maxy
so that I can pass it to a shapely function in the below form
from shapely.geometry import box
minx,miny,maxx,maxy=1,2,3,4
b = box(minx,miny,maxx,maxy)

What's the best way to do it? by avoiding, Cuda enabled or not or other exceptions?

Output:
minx, miny, maxx, maxy = [int(t.item()) for t in tensors]

where tensors is the list of tensors.


Instruction: Dead Kernel after running torchvision.utils.make_grid(images)

Input:
To make it simple, I am following this tutorial provided by PyTorch to create a CNN.
However, it appears that when I'm running this particular code block with this respective line:
# show images
imshow(torchvision.utils.make_grid(images))

It somehow kills the kernel. Which confuses me because it is only a simple function.
I have also went through numerous SO posts that are related to PyTorch but all of them are not related to torchvision. The most similar issue I found was posted 5 months ago without answer.
Please let me know if I need to add on any information to make this question clearer.

Output:
The same thing happened to me. Running these commands in the python interpreter, I got an error that lead me to this:
Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized
when I added the following, it worked:
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'


Instruction: Federated reinforcement learning

Input:
I am implementing federated deep Q-learning by PyTorch, using multiple agents, each running DQN.
My problem is that when I use multiple replay buffers for agents, each appending experiences at the corresponding agent, two elements of experiences in each agent replay buffer, i. e., "current_state" and "next_state" becomes the same after the first time slot. I mean in each buffer, we see the same values for current states and the same values for next states.
I have included simplified parts of the code and results below. Whay is it changing the current states and next states already exixting in the buffer when doing append? Is there something wrong with defining the buffers as a global variable? or do you have another idea?
<<< time 0 and agent 0:
current_state[0] = [1,2]
next_state[0] = [11,12]
*** experience: (array([ 1., 2.]), 2.0, array([200]), array([ 11., 12.]), 0)
*** buffer: deque([(array([ 1., 2.]), 2.0, array([200]), array([ 11., 12.]), 0)], maxlen=10000)

<<< time 0 and agent 1:
current_state[1] = [3, 4]
next_state[1] = [13, 14]
*** experience: (array([ 3., 4.]), 4.0, array([400]), array([ 13., 14.]), 0)
*** buffer: deque([(array([ 1., 2.]), 4.0, array([400]), array([ 11., 12.]), 0)], maxlen=10000)

<<< time 1 and agent 0:
current_state = [11,12]
next_state[0] = [110, 120]
*** experience: (array([ 11., 12.]), 6.0, array([600]), array([ 110., 120.]), 0)
*** buffer: deque([(array([ 11., 12.]), 2.0, array([200]), array([ 110., 120.]), 0),(array([ 11., 12.]), 6.0, array([600]), array([ 110., 120.]), 0)], maxlen=10000)

<<< time 1 and agent 1:
current_state = [13, 14]
next_state[1] = [130, 140]
*** experience: (array([ 13., 14.]), 8.0, array([800]), array([ 130., 140.]), 0)
*** buffer: deque([(array([ 13., 14.]), 4.0, array([400]), array([ 130., 140.]), 0),(array([ 13., 14.]), 8.0, array([800]), array([ 130., 140.]), 0)], maxlen=10000)

class BasicBuffer:
def __init__(self, max_size):
    self.max_size = max_size
    self.buffer = deque(maxlen=10000)

def add(self, current_state, action, reward, next_state, done):
    ## """"Add a new experience to buffer.""""
    experience = (current_state, action, np.array([reward]), next_state, done)
    self.buffer.append(experience)

def DQNtrain(env, state_size, agent):
for time in range(time_max):
    for e in range(agents_numbers):
       current_state[e,:]
        next_state_edge[e, :]
        ## """"Add a new experience to buffer.""""
        replay_buffer_t[e].add(current_state, action, reward, next_state, done)
        current_state[e, :] = next_state[e, :]

if __name__ == '__main__':
   DQNtrain(env, state_size, agent)
   replay_buffer_t = [[] for _ in range(edge_max)]
   for e in range(edge_max):
       replay_buffer_t[e] = BasicBuffer(max_size=agent_buffer_size)

Output:
I just found what is causing the problem. I should have used copy.deepcopy() for experiences:
experience = copy.deepcopy((current_state, action, np.array([reward]), next_state, done))
self.buffer.append(experience)


Instruction: Finding the Hessian matrix of this function

Input:
Hi I have the following function:
sum from 1 to 5000 -log(1−(xi)^2) -log(1-(a_i)^t*x), where a_i is a random vector and we are trying to minimize this function's value via Netwon's method.
I need a way to calculate the Hessian matrix with respect to (x1, x2, x3, ...). I tried auto-gradient but it took too much time. Here is my current time.
from autograd import elementwise_grad as egrad
from autograd import jacobian
import autograd.numpy as np

x=np.zeros(5000);
a = np.random.rand(5000,5000)
def f (x):
  sum = 0;
  for i in range(5000):
      sum += -np.log(1 - x[i]*x[i]) - np.log(1-np.dot(x,a[i]))

  return sum;

df = egrad(f)
d2f = jacobian(egrad(df));
print(d2f(x));

I have tried looking into sympy but I am confused on how to proceed.

Output:
PyTorch has a GPU optimised hessian operation:
import torch

torch.autograd.functional.hessian(func, inputs)


Instruction: What does the 1 in torch.Size([64, 1, 28, 28]) mean when I check a tensor shape?

Input:
I'm following this tutorial on towardsdatascience.com because I wanted to try the MNIST dataset using Pytorch since I've already done it using keras.
So in Step 2, knowing the dataset better, they print the trainloader's shape and it returns torch.Size([64, 1, 28, 28]). I understand  that 64 is the number of images in that loader and that each one is a 28x28 image but what does the 1 mean exactly?

Output: It simply defines an image of size 28x28 has 1 channel, which means it's a grayscale image. If it was a colored image then instead of 1 there would be 3 as the colored image has 3 channels such as RGB.


Instruction: Output evaluation loss after every n-batches instead of epochs with pytorch

Input:
Instead of printing the evaluation loss every epoch I would like to output it after every n-batches.
I have around 150'000 batches per epoch. I would like to output the evaluation loss every 50'000 batches.
Is this even possible? I am using pytorch and a pretrained bert model from huggingface.
My train loop:
best_valid_loss = float('inf')
train_losses=[]
valid_losses=[]

for epoch in range(params['epochs']):

      print('\n Epoch {:} / {:}'.format(epoch + 1, params['epochs']))

      #train model
      train_loss = train(scheduler, optimizer)

      #evaluate model
      valid_loss = evaluate()

      #save the best model
      if valid_loss < best_valid_loss:
          best_valid_loss = valid_loss
          torch.save(model.state_dict(), model_file)

      # append training and validation loss
      train_losses.append(train_loss)
      valid_losses.append(valid_loss)

      print(f'\nTraining Loss: {train_loss:.3f}')
      print(f'Validation Loss: {valid_loss:.3f}')

Output:
If you want to skip n elements of, for example, some list, you can do this using enumerate:
n = 50000

for i,epoch in enumerate(some_list):
     if i%n == 0:

           print('\n Epoch {:} / {:}'.format(epoch + 1, params['epochs']))
           ...

But in your case, you can use only an additional condition:
n = 50000

for epoch in range(params['epochs']):
     #train model
     train_loss = train(scheduler, optimizer)

     #evaluate model
     valid_loss = evaluate()

     #save the best model
     if valid_loss < best_valid_loss:
           best_valid_loss = valid_loss
           torch.save(model.state_dict(), model_file)

     # append training and validation loss
     train_losses.append(train_loss)
     valid_losses.append(valid_loss)

     ##
     # print every n cycles
     if epoch%n == 0:
           print('\n Epoch {:} / {:}'.format(epoch + 1, params['epochs']))

           print(f'\nTraining Loss: {train_loss:.3f}')
           print(f'Validation Loss: {valid_loss:.3f}')


Instruction: PyTorch tensors topk for every tensor across a dimension

Input:
I have the following tensor
inp = tensor([[[ 0.0000e+00,  5.7100e+02, -6.9846e+00],
     [ 0.0000e+00,  4.4070e+03, -7.1008e+00],
     [ 0.0000e+00,  3.0300e+02, -7.2226e+00],
     [ 0.0000e+00,  6.8000e+01, -7.2777e+00],
     [ 1.0000e+00,  5.7100e+02, -6.9846e+00],
     [ 1.0000e+00,  4.4070e+03, -7.1008e+00],
     [ 1.0000e+00,  3.0300e+02, -7.2226e+00],
     [ 1.0000e+00,  6.8000e+01, -7.2777e+00]],

    [[ 0.0000e+00,  2.1610e+03, -7.0754e+00],
     [ 0.0000e+00,  6.8000e+01, -7.2259e+00],
     [ 0.0000e+00,  1.0620e+03, -7.2920e+00],
     [ 0.0000e+00,  2.9330e+03, -7.3009e+00],
     [ 1.0000e+00,  2.1610e+03, -7.0754e+00],
     [ 1.0000e+00,  6.8000e+01, -7.2259e+00],
     [ 1.0000e+00,  1.0620e+03, -7.2920e+00],
     [ 1.0000e+00,  2.9330e+03, -7.3009e+00]],

    [[ 0.0000e+00,  4.4070e+03, -7.1947e+00],
     [ 0.0000e+00,  3.5600e+02, -7.2958e+00],
     [ 0.0000e+00,  3.0300e+02, -7.3232e+00],
     [ 0.0000e+00,  1.2910e+03, -7.3615e+00],
     [ 1.0000e+00,  4.4070e+03, -7.1947e+00],
     [ 1.0000e+00,  3.5600e+02, -7.2958e+00],
     [ 1.0000e+00,  3.0300e+02, -7.3232e+00],
     [ 1.0000e+00,  1.2910e+03, -7.3615e+00]]])

of shape
torch.Size([3, 8, 3])

and I would like to find the topk(k=4) elements across dim1, where the value to sort by is dim2 (the negative values). The resulting tensor shape should then be:
torch.Size([3, 4, 3])

I know how to do topk for a single tensor, but how do I do this for several batches at once?

Output:
I did it like this:
val, ind = inp[:, :, 2].squeeze().topk(k=4, dim=1, sorted=True)
new_ind = ind.unsqueeze(-1).repeat(1,1,3)
result = inp.gather(1, new_ind)

I don't know if this is the best way to do this but it worked.


Instruction: How is PyTorch's Class BCEWithLogitsLoss exactly implemented?

Input:
According to the PyTorch documentation, the advantage of the class BCEWithLogitsLoss() is that one can use the

log-sum-exp trick for numerical stability.

If we use the class BCEWithLogitsLoss() with the parameter reduction set to None, they have a formula for that:

I now simplified the terms, and obtain after some lines of calculation:

I was curious to see whether this is the way how the Source code does it, but I couldn't find it.. The only code they have is this:
Code for BCEWithLogitsLoss

Output:
nn.BCEWithLogitsLoss is actually just cross entropy loss that comes inside a sigmoid function. It may be used in case your model's output layer is not wrapped with sigmoid. Typically used with the raw output of a single output layer neuron.
Simply put, your model's output say pred will be a raw value. In order to get probability, you will have to use torch.sigmoid(pred). (To get actual class labels, you need torch.round(torch.sigmoid(pred)).) However, you don't need to do anything like that (i.e take sigmoid) when you use nn.BCEWithLogitsLoss. Here you just have to do the following-
criterion = nn.BCEWithLogitsLoss()
loss = criterion(pred, target) # pred is just raw nn output

Hence, coming to implementation part, criterion accepts two torch tensors - one being the raw nn outputs, the other being the true class labels, then wraps the first using sigmoid - for each element in the tensor and then calculates Cross Entropy loss (-(target*log(sigmoid(pred))) for each pair and reduces it to mean.


Instruction: Confused on using dropout in batch gradient descent with Q-learning

Input:
I am using PyTorch and adding dropout layers to my inner layers.
class MLP(nn.Module):
  #def __init__(self, n_inputs, n_action, n_hidden_layers=2, hidden_dim=8, drop=0.25):
  def __init__(self, console_file, n_inputs, n_action, layers_list, drop=0.25):
    super(MLP, self).__init__()
    print("Layers structure:")
    console_file.write("Layers structure:\n")
    print(f"inputs: {n_inputs}")
    console_file.write(f"inputs: {n_inputs}\n")
    self.layers = []
    for i, layer_size in enumerate(layers_list):
      if i == 0:
        layer = nn.Linear(n_inputs, layer_size)
      else:
        layer = nn.Linear(layers_list[i-1], layer_size)
      self.layers.append(layer)
      print(f"layer {i}: {layer_size}")
      console_file.write(f"layer {i}: {layer_size}\n")
      self.layers.append(nn.LeakyReLU(0.1))
      if drop > 0.01:
        #self.layers.append(nn.Dropout(p = drop**(len(layers_list)-i)))
        self.layers.append(nn.Dropout(p = drop))
        #print(f"drop {i}: {drop**(len(layers_list)-i)}")
        print(f"drop {i}: {drop}")
        #console_file.write(f"drop {i}: {drop**(len(layers_list)-i)}\n")
        console_file.write(f"drop {i}: {drop}\n")
    # final layer
    self.layers.append(nn.Linear(layers_list[-1], n_action))
    self.layers = nn.Sequential(*self.layers)
    print(f"outputs: {n_action}")
    console_file.write(f"outputs: {n_action}\n")
    print("========= NN structure =========\n")
    console_file.write("========= NN structure =========\n\n")
  def forward(self, X):
    return self.layers(X)

  def save_weights(self, path):
    torch.save(self.state_dict(), path)

  def load_weights(self, path):
    self.load_state_dict(torch.load(path))

I am making sure to turn training on during training and eval mode outside training (removing the droput layers).
    self.model = he.MLP(console_file, state_size, self.action_size, DIMENSION, DROPOUT)
    if DROPOUT > 0.01:
      self.model.train()
...
      if DROPOUT > 0.01: # before testing
        agent.model.eval()
...
    if DROPOUT > 0.01: # after testing
      self.model.train()

My confusion is as to HOW if at all PyTorch keeps track of which neurons it disables between the forward propagation and the later time where a random batch of output-and-reward is selected and backpropagation is performed on a specific case originating in a forward propagation with specific neurons disabled.
def train_one_step(model, criterion, optimizer, inputs, targets):
  # convert to tensors
  inputs = torch.from_numpy(inputs.astype(np.float32))
  targets = torch.from_numpy(targets.astype(np.float32))

  # zero the parameter gradients
  optimizer.zero_grad()

  # Forward pass
  outputs = model(inputs)
  loss = criterion(outputs, targets)

  # Backward and optimize
  loss.backward()
  optimizer.step()

inputs and targets are from random selection (batch over buffered history)
For me it makes sense to have the SAME neurons disabled between the forward and back propagation but since I can find nothing on the subject everything would suggest that the dropout will apply randomly and therefor nonsensically to me.
Either the forward and back propagations need to happen identically (and somehow PyTorch manages to remember by some markers, because I don't seem to pass any markers during batching) or I need to understand why they can be randomly different.

Output:
I'm not sure what is the problem, but let me try to explain how things work.
The .train() and .eval() calls only change the .training flag to True or False.
The Dropout layer samples the noise during the forward pass. Here's an example of forward implementation (I removed the ifs for the alpha and feature dropouts for readability):
template<bool feature_dropout, bool alpha_dropout, bool inplace, typename T>
Ctype<inplace> _dropout_impl(T& input, double p, bool train) {
  TORCH_CHECK(p >= 0 && p <= 1, "dropout probability has to be between 0 and 1, but got ", p);
  if (p == 0 || !train || input.numel() == 0) {
    return input;
  }

  if (p == 1) {
    return multiply<inplace>(input, at::zeros({}, input.options()));
  }

  auto noise = at::empty_like(input, LEGACY_CONTIGUOUS_MEMORY_FORMAT);
  noise.bernoulli_(1 - p);
  noise.div_(1 - p);
  return multiply<inplace>(input, noise);
}

As you can see, if !train (i.e., .eval()), it will return the input as it is. Moreover, you could say that it "remembers" which neurons were disabled the same way it "remembers" what is the value of each active neuron. Notice that the dropout layer actually works as a mask of 0s and (scaled) 1s on the output of the previous layer. It does not actually mask the neurons, although in pratice the effect is equivalent, since the neurons that generated the outputs multiplied by 0 will get no gradient, and the rest will get properly scaled gradients (because of the .div_(1-p)).


Instruction: Saving a pytoch tensor as a 32-bit grayscale Image

Input:
I have manipulated a 32-bit grayscale .tif image which I converted to tensor using PIL. After this I saved it with:
torchvision.utils.save_image(train_img_poac,fp=str(j)+".tif")

This method automatically converts the tensor to an RGB format image. I want my output image to be a 32-bit grayscale image.
I tried to use the arguments in the save_image function but could not find anything. Is converting it to numpy ndarray and then converting it to a 32-bit Image an option?

Output:
Unfortunately save_image doesn't have an option for preserving one-channel images. You can use a different library like OpenCV:
import cv2

image = train_img_poac.numpy()
cv2.imwrite('image_name', image)


Instruction: AttributeError:module 'torchtext.data' has no attribute 'TabularDataset'

Input:
I want to create a dataset from a tsv file with pytorch.
I was thinking of using
torchtext.data.TabularDataset.splits

but I'm getting an error message.
AttributeError:module 'torchtext.data' has no attribute 'TabularDataset'

Output: Try torchtext.legacy.data.TabularDataset.splits


Instruction: "module 'torchtext.data' has no attribute 'Field'"

Input:
import torchtext

ENGLISH = torchtext.data.Field(tokenize=tokenizer_english, lower=True, init_token="<sos>", eos_token="<eos>")

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-2a3d11c77e7d> in <module>
----> 1 ENGLISH = torchtext.data.Field(tokenize=tokenizer_english, lower=True, init_token="<sos>", eos_token="<eos>")

AttributeError: module 'torchtext.data' has no attribute 'Field'

It won't import torchtext.data.Field for some reason even though that's whats in the docs

Output:
[BC Breaking] Legacy
In v0.9.0 release, we move the following legacy code to torchtext.legacy. This is part of the work to revamp the torchtext library and the motivation has been discussed in Issue #664:

torchtext.legacy.data.field
torchtext.legacy.data.batch
torchtext.legacy.data.example
torchtext.legacy.data.iterator
torchtext.legacy.data.pipeline
torchtext.legacy.datasets

We have a migration tutorial to help users switch to the torchtext datasets in v0.9.0 release. For the users who still want the legacy components, they can add legacy to the import path.

Try it with ENGLISH = torchtext.legacy.data.field(tokenize=tokenizer_english, lower=True, init_token="<sos>", eos_token="<eos>")


Instruction: In Pytorch, is there a difference between (x<0) and x.lt(0)?

Input:
Suppose x is a tensor in Pytorch. One can either write:
x_lowerthanzero = x.lt(0)

or:
x_lowerthanzero = (x<0)

with seemingly the exact same results. Many other operations have Pytorch built-in equivalents: x.gt(0) for (x>0), x.neg() for -x, x.mul() etc.
Is there a good reason to use one form over the other?

Output:
They are equivalent. < is simply a more readable alias.
Python operators have canonical function mappings e.g:
Algebraic operations

Operation
Syntax
Function

Addition
a + b
add(a, b)

Subtraction
a - b
sub(a, b)

Multiplication
a * b
mul(a, b)

Division
a / b
truediv(a, b)

Exponentiation
a ** b
pow(a, b)

Matrix Multiplication
a @ b
matmul(a, b)

Comparisons

Operation
Syntax
Function

Ordering
a < b
lt(a, b)

Ordering
a <= b
le(a, b)

Equality
a == b
eq(a, b)

Difference
a != b
ne(a, b)

Ordering
a >= b
ge(a, b)

Ordering
a > b
gt(a, b)

You can check that these are indeed mapped to the respectively named torch functions here e.g:
def __lt__(self, other):
    return self.lt(other)


Instruction: ValueError: Unknown CUDA arch (8.6) or GPU not supported

Input:
when I build DCNv2 in my conda environment,I got this message.
I have checked cuda by nvidia-smi:
Tue Apr  6 20:03:13 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.67       Driver Version: 460.67       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3070    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   45C    P8    17W / 220W |    448MiB /  7979MiB |     23%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       955      G   /usr/lib/xorg/Xorg                 53MiB |
|    0   N/A  N/A      1555      G   /usr/lib/xorg/Xorg                210MiB |
|    0   N/A  N/A      1690      G   /usr/bin/gnome-shell               61MiB |
|    0   N/A  N/A      3564      G   ...AAAAAAAAA= --shared-files      108MiB |

and nvcc -V:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Any help?

Output: Your GPU is "too new" for CUDA 10. Support for cards with compute capability 8.6 like yours was added in CUDA 11.1 (thank you @RobertCrovella for the correction). This means you'll need to use CUDA 11.1 or newer.


Instruction: how to change the labels in a datafolder of pytorch?

Input:
I first load an unlabeled dataset as following:
unlabeled_set = DatasetFolder("food-11/training/unlabeled", loader=lambda x: Image.open(x), extensions="jpg", transform=train_tfm)
and now since I'm trying to conduct semi-supervised learning: I'm trying to define the following function. The input "dataset" is the unlabeled_set I just loaded.
As I want to change the label of the dataset to be the one I predicted, not the original labels(all of the original labels were 1's), how can I do?
I have tried using dataset.targets to change the labels, but it doesn't work at all.
the following is my function:
import torch
def get_pseudo_labels(dataset, model, threshold=0.07):
    # This functions generates pseudo-labels of a dataset using given model.
    # It returns an instance of DatasetFolder containing images whose prediction confidences exceed a given threshold.
    # You are NOT allowed to use any models trained on external data for pseudo-labeling.
    device = "cuda" if torch.cuda.is_available() else "cpu"
    x = []
    y = []

    # print(dataset.targets[0])

    # Construct a data loader.
    data_loader = DataLoader(dataset, batch_size=batch_size, shuffle=False)

    # Make sure the model is in eval mode.
    model.eval()
    # Define softmax function.
    softmax = nn.Softmax()
    counter = 0
    # Iterate over the dataset by batches.
    for batch in tqdm(data_loader):
        img, _ = batch

        # Forward the data
        # Using torch.no_grad() accelerates the forward process.
        with torch.no_grad():
            logits = model(img.to(device))

        # Obtain the probability distributions by applying softmax on logits.
        probs = softmax(logits)
        count = 0
        # ---------- TODO ----------
        # Filter the data and construct a new dataset.
        dataset.targets = torch.tensor(dataset.targets)
        for p in probs:
          if torch.max(p) >= threshold:
            if not(counter in x):
              x.append(counter)
            dataset.targets[counter] = torch.argmax(p)

          counter += 1

    # Turn off the eval mode.
    model.train()
    # dat = DataLoader(ImgDataset(x,y), batch_size=batch_size, shuffle=False)
    print(dataset.targets[10])
    new = torch.utils.data.Subset(dataset, x)

    return new```

Output:
PyTorch DataSets can return tuples of values, but they have no inherent "features"/"target" distinction. You can create your modified DataSet like so:
labeled_data = [*zip(dataset, labels)]
data_loader = DataLoader(labeled_dataset, batch_size=batch_size, shuffle=False)

for imgs, labels in data_loader: # per batch
    ...


Instruction: PyTorch warning about using a non-full backward hook when the forward contains multiple autograd Nodes

Input:
After a recent upgrade, when running my PyTorch loop, I now get the warning

Using a non-full backward hook when the forward contains multiple autograd Nodes`".

The training still runs and completes, but I am unsure where I am supposed to place the register_full_backward_hook function.
I have tried adding it to each of the layers in my neural network but this gives further errors about using different hooks.
Can anyone please advise?

Output:
PyTorch version 1.8.0 deprecated register_backward_hook (source code) in favor of register_full_backward_hook (source code).
You can find it in the patch notes here: Deprecated old style nn.Module backward hooks (PR #46163)
The warning you're getting:

Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.

Simply indicates that you should replace all register_backward_hook calls with register_full_backward_hook in your code to get the behavior described in the documentation page.


Instruction: Installing PyTorch on Jetson Nano Ubuntu 18

Input:
I am trying to install PyTorch on Jetson Nano Ruining Ubuntu 1804. My reference is https://dev.to/evanilukhin/guide-to-install-pytorch-with-cuda-on-ubuntu-18-04-5217
When I try the following command this is what I get:
(my_env) crigano@crigano-desktop:~$ python3.8 -m pip install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing
Collecting numpy
  Using cached numpy-1.20.2-cp38-cp38-manylinux2014_aarch64.whl (12.7 MB)
Collecting ninja
  Using cached ninja-1.10.0.post2.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pyyaml
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux2014_aarch64.whl (818 kB)
ERROR: Could not find a version that satisfies the requirement mkl
ERROR: No matching distribution found for mkl

Output:
If you just want to use PyTorch on the bare-metal Jetson Nano, simply install it with NVIDIA's pre-compiled binary wheel. Other packages can be found in the Jetson Zoo.
MKL is developed by Intel "to optimize code for current and future generations of Intel® CPUs and GPUs." [PyPI]. Apparently it does run on other x86-based chips like AMD's (although Intel has historically intentionally crippled the library for non-Intel chips [Wikipedia]), but unsurprisingly Intel is not interested in supporting ARM devices and has not ported MKL to ARM architectures.
If your goal is to use MKL for math optimization in numpy, openblas is a working alternative for ARM. libopenblas-base:arm64 and libopenblas-dev:arm64 come pre-installed on NVIDIA's "L4T PyTorch" Docker images. You can confirm that numpy detects them with numpy.__config__.show(). This is what I get using numpy 1.12 in python 3.69 on the l4t-pytorch:r32.5.0-pth1.6-py3 image:
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib/aarch64-linux-gnu']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib/aarch64-linux-gnu']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib/aarch64-linux-gnu']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib/aarch64-linux-gnu']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

So presumably it will use openblas in place of MKL for math optimization. If your use case is also for numpy optimization, you can likewise use openblas and shouldn't need MKL... which is fortunate, since it isn't available anyway.


Instruction: Get a list of tensor from masked indices

Input:
I'm trying to get a list of tensors based on different group,
e.g.,
x = tensor([ 0.3018, -0.0079,  1.4995, -1.4422,  1.6007])

indices = torch.tensor([0,0,1,1,2])

res = func(x,indices)

I want my result to be
res= [[0.3018, -0.0079], [1.4995, -1.4422], [1.6007]]

I'm wondering how can I achieve this result, I checked gather and index_select,
but I can't get the result like above.
Thank you!

Output:
How about
res = [x[indices == i_] for i_ in indices.unique()]


Instruction: loading model failed in torchserving

Input:
i learning to serve a model using pytorch serving and i am new to this serving
this is the handler file i created for serving the vgg16 model
i am using the model from kaggle
Myhandler.py file

import io
import os
import logging
import torch
import numpy as np
import torch.nn.functional as F
from PIL import Image
from torchvision import transforms,datasets, models
from ts.torch_handler.image_classifier import ImageClassifier
from ts.torch_handler.base_handler import BaseHandler
from ts.utils.util import list_classes_from_module
import importlib
from torch.autograd import Variable
import seaborn as sns
import torchvision
from torch import optim, cuda
from torch.utils.data import DataLoader, sampler
import torch.nn as nn

import warnings
warnings.filterwarnings('ignore', category=FutureWarning)

# Data science tools
import pandas as pd

#path = 'C:\\Users\\fazil\\OneDrive\\Desktop\\pytorch\\vgg11\\vgg16.pt'

path = r'C:\Users\fazil\OneDrive\Desktop\pytorch\vgg11\vgg16.pt'
#image = r'C:\Users\fazil\OneDrive\Desktop\pytorch\vgg11\normal.jpeg'

class VGGImageClassifier(ImageClassifier):
    """
    Overriding the model loading code as a workaround for issue :
    https://github.com/pytorch/serve/issues/535
    https://github.com/pytorch/vision/issues/2473
    """
    def __init__(self):
        self.model = None
        self.mapping = None
        self.device = None
        self.initialized = False

    def initialize(self,context):
        """load eager mode state_dict based model"""
        properties = context.system_properties
        #self.device = torch.device(
        #"cuda:" + str(properties.get("gpu_id"))
        #if torch.cuda.is_available()
        #    else "cpu"
        #)
        model_dir = properties.get("model_dir")

        model_pt_path = os.path.join(model_dir, "model.pt")
        # Read model definition file
        model_def_path = os.path.join(model_dir, "model.py")
        if not os.path.isfile(model_def_path):
            raise RuntimeError("Missing the model definition file")

        checkpoint = torch.load(path, map_location='cpu')
        logging.error('%s ',checkpoint)
        self.model = models.vgg16(pretrained=True)
        logging.error('%s ',self.model)
        self.model.classifier = checkpoint['classifier']
        logging.error('%s ', self.model.classifier )
        self.model.load_state_dict(checkpoint['state_dict'], strict=False)
        self.model.class_to_idx = checkpoint['class_to_idx']
        self.model.idx_to_class = checkpoint['idx_to_class']
        self.model.epochs = checkpoint['epochs']

        optimizer = checkpoint['optimizer']
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])

        for param in model.parameters():
            param.requires_grad = False

        logger.debug('Model file {0} loaded successfully'.format(model_pt_path))
        self.initialized = True

    def preprocess(self,data):
        image = data.get("data")
        if image is None:
            image = data.get("body")
        image_transform =transforms.Compose([
        transforms.Resize(size=256),
        transforms.CenterCrop(size=224),
        transforms.ToTensor(),
        transforms.Normalize((0.5), (0.5))
        ])
        image = Image.open(io.BytesIO(image)).convert('RGB')
        image = image_transform(image)
        image = image.unsqueeze(0)
        return image

    def inference(self, image):

        outs = self.model.forward(image)
        probs = F.softmax(outs , dim=1)
        preds = torch.argmax(probs, dim=1)
        logging.error('%s ',preds)
        return preds

    def postprocess(self, preds):
        res = []

        preds = preds.cpu().tolist()
        for pred in preds:
            label = self.mapping[str(pred)] [1]
            res.append({'label': label , 'index': pred })
        return res

_service = VGGImageClassifier()

def handle(data,context):
    if not _service.initialized:
        _service.initialize(context)

    if data is None:
        return None
    data = _service.preprocess(data)
    data = _service.inference(data)
    data = _service.postprocess(data)

    return data

this is the error i got
Torchserve version: 0.3.1
TS Home: C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages
Current directory: C:\Users\fazil\OneDrive\Desktop\pytorch\vgg11
Temp directory: C:\Users\fazil\AppData\Local\Temp
Number of GPUs: 0
Number of CPUs: 4
Max heap size: 3038 M
Python executable: c:\users\fazil\anaconda3\envs\serve\python.exe
Config file: ./config.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8081
Metrics address: http://0.0.0.0:8082
Model Store: C:\Users\fazil\OneDrive\Desktop\pytorch\vgg11\model_store
Initial Models: vgg16.mar
Log dir: C:\Users\fazil\OneDrive\Desktop\pytorch\vgg11\logs
Metrics dir: C:\Users\fazil\OneDrive\Desktop\pytorch\vgg11\logs
Netty threads: 32
Netty client threads: 0
Default workers per model: 4
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: false
Metrics report format: prometheus
Enable metrics API: true
2021-04-08 12:33:22,517 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: vgg16.mar
2021-04-08 12:33:40,392 [INFO ] main org.pytorch.serve.archive.ModelArchive - eTag 85b61fc819804aea9db0ca8786c2e427
2021-04-08 12:33:40,423 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model vgg16
2021-04-08 12:33:40,424 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model vgg16
2021-04-08 12:33:40,424 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model vgg16 loaded.
2021-04-08 12:33:40,426 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: vgg16, count: 4
2021-04-08 12:33:40,481 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: NioServerSocketChannel.
2021-04-08 12:33:41,173 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: None
2021-04-08 12:33:41,177 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: None
2021-04-08 12:33:41,180 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]12328
2021-04-08 12:33:41,180 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]14588
2021-04-08 12:33:41,180 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2021-04-08 12:33:41,181 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2021-04-08 12:33:41,181 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.8
2021-04-08 12:33:41,181 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.8
2021-04-08 12:33:41,186 [DEBUG] W-9001-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-vgg16_1.0 State change null -> WORKER_STARTED
2021-04-08 12:33:41,186 [DEBUG] W-9002-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-vgg16_1.0 State change null -> WORKER_STARTED
2021-04-08 12:33:41,199 [INFO ] W-9001-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /127.0.0.1:9001
2021-04-08 12:33:41,199 [INFO ] W-9002-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /127.0.0.1:9002
2021-04-08 12:33:41,240 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: None
2021-04-08 12:33:41,244 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]12008
2021-04-08 12:33:41,244 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2021-04-08 12:33:41,245 [DEBUG] W-9000-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-vgg16_1.0 State change null -> WORKER_STARTED
2021-04-08 12:33:41,245 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.8
2021-04-08 12:33:41,245 [INFO ] W-9000-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /127.0.0.1:9000
2021-04-08 12:33:41,255 [INFO ] W-9003-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: None
2021-04-08 12:33:41,260 [INFO ] W-9003-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]15216
2021-04-08 12:33:41,260 [INFO ] W-9003-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2021-04-08 12:33:41,261 [DEBUG] W-9003-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-vgg16_1.0 State change null -> WORKER_STARTED
2021-04-08 12:33:41,261 [INFO ] W-9003-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.8
2021-04-08 12:33:41,262 [INFO ] W-9003-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /127.0.0.1:9003
2021-04-08 12:33:41,768 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080
2021-04-08 12:33:41,768 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: NioServerSocketChannel.
2021-04-08 12:33:41,774 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081
2021-04-08 12:33:41,775 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: NioServerSocketChannel.
2021-04-08 12:33:41,777 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://0.0.0.0:8082
2021-04-08 12:33:41,784 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: ('127.0.0.1', 9001).
2021-04-08 12:33:41,784 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: ('127.0.0.1', 9002).
2021-04-08 12:33:41,784 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: ('127.0.0.1', 9000).
2021-04-08 12:33:41,784 [INFO ] W-9003-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: ('127.0.0.1', 9003).
Model server started.
2021-04-08 12:33:48,486 [INFO ] pool-2-thread-1 TS_METRICS - CPUUtilization.Percent:100.0|#Level:Host|#hostname:fazil,timestamp:1617865428
2021-04-08 12:33:48,487 [INFO ] pool-2-thread-1 TS_METRICS - DiskAvailable.Gigabytes:74.49674987792969|#Level:Host|#hostname:fazil,timestamp:1617865428
2021-04-08 12:33:48,491 [INFO ] pool-2-thread-1 TS_METRICS - DiskUsage.Gigabytes:147.9403419494629|#Level:Host|#hostname:fazil,timestamp:1617865428
2021-04-08 12:33:48,496 [INFO ] pool-2-thread-1 TS_METRICS - DiskUtilization.Percent:66.5|#Level:Host|#hostname:fazil,timestamp:1617865428
2021-04-08 12:33:48,499 [INFO ] pool-2-thread-1 TS_METRICS - MemoryAvailable.Megabytes:4488.515625|#Level:Host|#hostname:fazil,timestamp:1617865428
2021-04-08 12:33:48,504 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUsed.Megabytes:7658.80859375|#Level:Host|#hostname:fazil,timestamp:1617865428
2021-04-08 12:33:48,513 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUtilization.Percent:63.0|#Level:Host|#hostname:fazil,timestamp:1617865428
2021-04-08 12:34:24,385 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2021-04-08 12:34:24,439 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2021-04-08 12:34:24,440 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 182, in <module>
2021-04-08 12:34:24,443 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2021-04-08 12:34:24,444 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 154, in run_server
2021-04-08 12:34:24,446 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2021-04-08 12:34:24,446 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 116, in handle_connection
2021-04-08 12:34:24,447 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2021-04-08 12:34:24,448 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 89, in load_model
2021-04-08 12:34:24,523 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service = model_loader.load(model_name, model_dir, handler, gpu, batch_size, envelope)
2021-04-08 12:34:24,582 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "c:\users\fazil\anaconda3\envs\serve\lib\site-packages\ts\model_loader.py", line 104, in load
2021-04-08 12:34:24,597 [INFO ] nioEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2021-04-08 12:34:24,583 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2021-04-08 12:34:24,646 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     initialize_fn(service.context)
2021-04-08 12:34:24,646 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2021-04-08 12:34:24,649 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 182, in <module>
2021-04-08 12:34:24,649 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2021-04-08 12:34:24,650 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 154, in run_server
2021-04-08 12:34:24,648 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "c:\users\fazil\anaconda3\envs\serve\lib\site-packages\ts\model_loader.py", line 131, in <lambda>
2021-04-08 12:34:24,652 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2021-04-08 12:34:24,649 [DEBUG] W-9000-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2021-04-08 12:34:24,734 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 116, in handle_connection
2021-04-08 12:34:24,653 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     initialize_fn = lambda ctx: entry_point(None, ctx)
2021-04-08 12:34:24,734 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2021-04-08 12:34:24,735 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\AppData\Local\Temp\models\85b61fc819804aea9db0ca8786c2e427\hanndler.py", line 268, in handle
2021-04-08 12:34:24,735 [DEBUG] W-9000-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
        at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
        at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:188)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2021-04-08 12:34:24,736 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 89, in load_model
2021-04-08 12:34:24,753 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service = model_loader.load(model_name, model_dir, handler, gpu, batch_size, envelope)
2021-04-08 12:34:24,736 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     _service.initialize(context)
2021-04-08 12:34:24,754 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\AppData\Local\Temp\models\85b61fc819804aea9db0ca8786c2e427\hanndler.py", line 111, in initialize
2021-04-08 12:34:24,754 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "c:\users\fazil\anaconda3\envs\serve\lib\site-packages\ts\model_loader.py", line 104, in load
2021-04-08 12:34:24,754 [WARN ] W-9000-vgg16_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: vgg16, error: Worker died.
2021-04-08 12:34:24,756 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     initialize_fn(service.context)
2021-04-08 12:34:24,755 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.model.classifier = checkpoint['classifier']
2021-04-08 12:34:24,758 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "c:\users\fazil\anaconda3\envs\serve\lib\site-packages\ts\model_loader.py", line 131, in <lambda>
2021-04-08 12:34:24,810 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     initialize_fn = lambda ctx: entry_point(None, ctx)
2021-04-08 12:34:24,811 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\AppData\Local\Temp\models\85b61fc819804aea9db0ca8786c2e427\hanndler.py", line 268, in handle
2021-04-08 12:34:24,757 [DEBUG] W-9000-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-vgg16_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2021-04-08 12:34:24,871 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     _service.initialize(context)
2021-04-08 12:34:24,872 [WARN ] W-9000-vgg16_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-vgg16_1.0-stderr
2021-04-08 12:34:24,812 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - KeyError: 'classifier'
2021-04-08 12:34:24,872 [WARN ] W-9000-vgg16_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-vgg16_1.0-stdout
2021-04-08 12:34:24,872 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\AppData\Local\Temp\models\85b61fc819804aea9db0ca8786c2e427\hanndler.py", line 111, in initialize
2021-04-08 12:34:24,874 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.model.classifier = checkpoint['classifier']
2021-04-08 12:34:24,903 [INFO ] nioEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9001 Worker disconnected. WORKER_STARTED
2021-04-08 12:34:24,876 [INFO ] W-9000-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds.
2021-04-08 12:34:24,931 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - KeyError: 'classifier'
2021-04-08 12:34:24,932 [DEBUG] W-9001-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2021-04-08 12:34:24,974 [DEBUG] W-9001-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
        at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
        at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:188)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2021-04-08 12:34:25,015 [WARN ] W-9001-vgg16_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: vgg16, error: Worker died.
2021-04-08 12:34:25,015 [DEBUG] W-9001-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-vgg16_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2021-04-08 12:34:25,016 [WARN ] W-9001-vgg16_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9001-vgg16_1.0-stderr
2021-04-08 12:34:25,017 [WARN ] W-9001-vgg16_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9001-vgg16_1.0-stdout
2021-04-08 12:34:25,017 [INFO ] W-9001-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9001 in 1 seconds.
2021-04-08 12:34:25,038 [INFO ] W-9000-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-vgg16_1.0-stdout
2021-04-08 12:34:25,038 [INFO ] W-9000-vgg16_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-vgg16_1.0-stderr
2021-04-08 12:34:25,085 [INFO ] W-9001-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9001-vgg16_1.0-stdout
2021-04-08 12:34:25,085 [INFO ] W-9001-vgg16_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9001-vgg16_1.0-stderr
2021-04-08 12:34:25,247 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2021-04-08 12:34:25,247 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2021-04-08 12:34:25,248 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 182, in <module>
2021-04-08 12:34:25,248 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2021-04-08 12:34:25,248 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 154, in run_server
2021-04-08 12:34:25,249 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2021-04-08 12:34:25,250 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 116, in handle_connection
2021-04-08 12:34:25,250 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2021-04-08 12:34:25,251 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\anaconda3\envs\serve\Lib\site-packages\ts\model_service_worker.py", line 89, in load_model
2021-04-08 12:34:25,251 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service = model_loader.load(model_name, model_dir, handler, gpu, batch_size, envelope)
2021-04-08 12:34:25,253 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "c:\users\fazil\anaconda3\envs\serve\lib\site-packages\ts\model_loader.py", line 104, in load
2021-04-08 12:34:25,253 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     initialize_fn(service.context)
2021-04-08 12:34:25,254 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "c:\users\fazil\anaconda3\envs\serve\lib\site-packages\ts\model_loader.py", line 131, in <lambda>
2021-04-08 12:34:25,254 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     initialize_fn = lambda ctx: entry_point(None, ctx)
2021-04-08 12:34:25,254 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\AppData\Local\Temp\models\85b61fc819804aea9db0ca8786c2e427\hanndler.py", line 268, in handle
2021-04-08 12:34:25,255 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     _service.initialize(context)
2021-04-08 12:34:25,256 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "C:\Users\fazil\AppData\Local\Temp\models\85b61fc819804aea9db0ca8786c2e427\hanndler.py", line 111, in initialize
2021-04-08 12:34:25,257 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.model.classifier = checkpoint['classifier']
2021-04-08 12:34:25,257 [INFO ] W-9002-vgg16_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - KeyError: 'classifier'
2021-04-08 12:34:25,454 [INFO ] nioEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9002 Worker disconnected. WORKER_STARTED
2021-04-08 12:34:25,456 [DEBUG] W-9002-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2021-04-08 12:34:25,457 [DEBUG] W-9002-vgg16_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
        at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
        at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:188)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2021-04-08 12:34:25,482 [WARN ] W-9002-vgg16_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: vgg16, error: Worker died.

and also i load the model from path
because i got error if i use model_pt_path
can some one help me with this

Output:
i am using the model from kaggle

I presume you got the model from https://www.kaggle.com/pytorch/vgg16
I think you are loading the model incorrectly.
You are loading a checkpoint, which would work if your model was saved like this:
torch.save({
        'epoch': epoch,
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict(),
        'loss': loss,
        ...
        }, PATH)

But it was probably saved like this:
torch.save(model.state_dict(), PATH)

Which would explain the KeyError.
I modified the initialize method according to the second case:
def initialize(self,context):
    """load eager mode state_dict based model"""
    properties = context.system_properties
    model_dir = properties.get("model_dir")

    model_pt_path = os.path.join(model_dir, "model.pt")
    # Read model definition file
    model_def_path = os.path.join(model_dir, "model.py")
    if not os.path.isfile(model_def_path):
        raise RuntimeError("Missing the model definition file")

    state_dict = torch.load(path, map_location='cpu')
    # logging.error('%s ',checkpoint)
    self.model = models.vgg16(pretrained=True)
    logging.error('%s ',self.model)
    # self.model.classifier = checkpoint['classifier']
    # logging.error('%s ', self.model.classifier )
    self.model.load_state_dict(state_dict, strict=False)
    # self.model.class_to_idx = checkpoint['class_to_idx']
    # self.model.idx_to_class = checkpoint['idx_to_class']
    # self.model.epochs = checkpoint['epochs']

    # optimizer = checkpoint['optimizer']
    # optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    # for param in model.parameters():
    #     param.requires_grad = False
    # logger.debug('Model file {0} loaded successfully'.format(model_pt_path))
    self.initialized = True

Using the model linked above, I managed to start torchserve without error.


Instruction: PyTorch input/output data sizes

Input:
I'm trying out PyTorch for the first time, and running into a couple problems. I've shared some of my code below and have two questions.
Q1: What should my output size be? Each input should lead to one output, equal to one of 6 possible output labels (1-6). Should output size be 1 or 6?
Q2: Something is wrong with my accuracy calculation, which I think is tied into Q1. predicted ends up being of size 4 x 4411, where 4 is my batch size (and so I think that's correct) but 4411 is my feature/input size, which is almost certainly wrong. I would expect it to be 6 (number of possible output labels). labels is 4x6, which I think is correct. If I change the dim I'm taking the max over from 1 to 2, then it gets the correct size of 4x6, but logically it makes no sense as it's returning the max index across all feature values for an input.
I think I'm missing something crucial about what pytorch is doing with my data. I feel so close.... Any ideas on how I can fix this? Thanks!
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(Net, self).__init__()
        self.input_size  = input_size
        self.output_size = output_size
        self.hidden_dim  = hidden_dim
        self.n_layers    = n_layers

        self.rnn = torch.nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
        # Q1: What should my output_size be?
        self.fc = torch.nn.Linear(hidden_dim, output_size)

    def forward(self, x):
        batch_size = x.size(0)
        hidden = self.init_hidden(batch_size)
        out, hidden = self.rnn(x, hidden)
        out = self.fc(out)
        return out, hidden

    def init_hidden(self, batch_size):
        return torch.zeros(self.n_layers, batch_size, self.hidden_dim)
if __name__ == '__main__':
   # ... code removed that just creates the Dataloaders, and initialises some size variables

    net = Net(input_size=dataset.input_size, output_size=6, hidden_dim=24, n_layers=1)
    net.to(device)
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(net.parameters(), lr=lr)

    for epoch in range(1, epochs+1):
        total = 0
        correct = 0
        for inputs, labels in dataloaders['train']:
            optimizer.zero_grad()
            inputs, labels = inputs.to(device), labels.to(device)
            output, hidden = net(inputs)
            loss = criterion(output, labels) # error
            loss.backward()
            optimizer.step()

            # Q2: Something is wrong here
            _, predicted = torch.max(output.data, 1)
            total += inputs.size(0)
            correct += (predicted == labels).sum().item()
            print(predicted == labels)
        accuracy = 100 * correct / total```

Output:
Pytorch's CrossEntropyLoss expects output of size (n) (with the score of each of the n classes) and label as an integer of the correct class's index.
Your rnn based model is spitting out tensors of shape [batch, input_size, 6], since it is an rnn and producing a sequence of the same length as the input (with 6 scores per element of the sequence). If you wish to have one class label prediction per sequence (not per element of sequence) you will need to collapse this to a tensor of shape [batch, 6].


Instruction: Pytorch: how to change requires_grad to be true in an OrderedDict

Input:
Suppose I have a neural network object from torch.nn, by default the requires_grad is False for its parameters. I want to change it to be True. But the following naive approach fails:
From torch import nn
a = nn.Linear(1, 1)
a.state_dict()[‘weight’].requires_grad = True
print(a.state_dict()[‘weight’].requires_grad)

The result is False. Could anyone explain what the problem is and how to fix it? Thank you! My torch version is 1.7.1.

Output:
By default trainable nn objects parameters will have requires_grad=True.
You can verify that by doing:
import torch.nn as nn

layer = nn.Linear(1, 1)

for param in layer.parameters():
    print(param.requires_grad)

# or use
print(layer.weight.requires_grad)
print(layer.bias.requires_grad)

To change requires_grad state:
for param in layer.parameters():
    param.requires_grad = False # or True


Instruction: PyTorch - Change weights of Conv2d

Input:
For some reason, I cannot seem to assign all the weights of a Conv2d layer in PyTorch - I have to do it in two steps. Can anyone help me with what I am doing wrong?
layer = torch.nn.Conv2d(in_channels=1, out_channels=2, kernel_size=(2,2), stride=(2,2))
layer.state_dict()['weight']

gives me a tensor of size (2,1,2,2)
tensor([[[[ 0.4738, -0.2197],
      [-0.3436, -0.0754]]],

    [[[ 0.1662,  0.4098],
      [-0.4306, -0.4828]]]])

When I try to assign weights like so
layer.state_dict()['weight'] = torch.tensor([
  [[[ 1, 2],
    [3, 4]]],

  [[[-1, -2],
    [-3, -4]]]
])

the weights don't change. However, if I do something like this
layer.state_dict()['weight'][0] = torch.tensor([
  [[[1, 2],
    [3, 4]]],
])
layer.state_dict()['weight'][1] = torch.tensor([
  [[[-1, -2],
    [-3, -4]]],
])

The weights change. Why is this so?

Output:
I'm not sure about why you can't directly assign them but the more proper way to achieve what you're trying to do would be
layer.load_state_dict({'weight': torch.tensor([[[[0.4738, -0.2197],
                      [-0.3436, -0.0754]]],
                    [[[0.1662, 0.4098],
                      [-0.4306, -0.4828]]]])}, strict=False)


Instruction: Different test results with pytorch lightning

Input:
I use Pytorch Lightning to train a small NN transfert learning) with the hymenoptera photos (inspired from here).
In the test_step method, it prints the real classes (classes) and the predictions (preds).
After the training, I do the same (verification step) but I get different results.
import torch
from torch import nn
from torch.optim import Adam,SGD
import pytorch_lightning as pl
from torchvision import  models
from torch.optim import lr_scheduler
from pytorch_lightning.metrics.functional import accuracy
from pytorch_lightning.loggers import TensorBoardLogger
from hymenoptereDataModule import HymenopteraDataModule

class LitHymenoptera(pl.LightningModule):

    def __init__(self, batch_size=4):
        super().__init__()
        torch.manual_seed(42)
        self.batch_size = batch_size
        self.dataModule = HymenopteraDataModule()
        self.dataModule.setup()
        self.criterion = nn.CrossEntropyLoss()
        self.logger = TensorBoardLogger('tb_logs', name=f'Model')
        # Define the model
        self.model = models.resnet18(pretrained=True)
        num_ftrs = self.model.fc.in_features
        self.model.fc = nn.Linear(num_ftrs, 2)

    def forward(self, x):
        return self.model(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        logits = self.model(x)
        # Compute loss
        loss = self.criterion(logits, y)
        # training metrics
        preds = torch.argmax(logits, dim=1)
        acc = accuracy(preds, y)
        num_correct = torch.eq(preds.view(-1), y.view(-1)).sum()
        return {'loss': loss,
                'acc': acc,
                'num_correct': num_correct}

    def training_epoch_end(self, outputs):
        self.exp_lr_scheduler.step()

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self.model(x)
        loss = self.criterion(logits, y)
        # validation metrics
        preds = torch.argmax(logits, dim=1)
        acc = accuracy(preds, y)
        num_correct = torch.eq(preds.view(-1), y.view(-1)).sum()
        return {'loss': loss,
                'acc': acc,
                'num_correct': num_correct}

    def test_step(self, batch, batch_idx):
        inputs, classes = batch
        logits = self(inputs)
        preds = torch.argmax(logits, dim=1)
        print('###############################')
        print('classes1 = ',classes)
        print('preds1 = ',preds)
        print(logits)

    def configure_optimizers(self):
        optimizer = SGD(self.parameters(), lr=0.001, momentum=0.9)
        # Decay LR by a factor of 0.1 every 7 epochs
        self.exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
        return optimizer

model = LitHymenoptera()
trainer = pl.Trainer(gpus=1, max_epochs=5, progress_bar_refresh_rate=100)
trainer.fit(model,model.dataModule)
trainer.test(model)

# Now, another test
for inputs, classes in model.dataModule.val_dataloader():
    print('###############################')
    logits = model(inputs.cuda())
    preds = torch.argmax(logits, dim=1)
    print('classes2 = ', classes)
    print('preds2 = ', preds)
    print(logits)

Here's the first output from test_step method :
classes1 =  tensor([0, 0, 0, 0], device='cuda:0')
preds1 =  tensor([1, 0, 0, 0], device='cuda:0') tensor([[0.1626, 0.2195],
            [1.1437, 0.5745],
            [0.9351, 0.4271],
            [0.7365, 0.5342]], device='cuda:0')

and now the first output from the verification step :
classes2 =  tensor([0, 0, 0, 0])
preds2 =  tensor([1, 0, 1, 0], device='cuda:0')
tensor([[-0.0168,  0.0800],
        [ 0.6817,  0.2949],
        [-0.2205,  0.1009],
        [ 0.6126,  0.4924]], device='cuda:0', grad_fn=<AddmmBackward>)

Both classes are identical (and I check the images, they are the same) but the preds are different.
Where does it come from?

Output:
I realize that I forget to add:
model.freeze()

before using the model for the second time.
So, now, both results are the same.


Instruction: Detectron2 Speed up inference instance segmentation

Input:
I have working instance segmentation, I'm using "mask_rcnn_R_101_FPN_3x" model. When I inference image it takes about 3 second / image on GPU. How can I speed up it faster ?
I code in Google Colab
This is my setup config:
cfg = get_cfg()

cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))

cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1

cfg.OUTPUT_DIR = "/content/drive/MyDrive/TEAM/save/"

cfg.DATASETS.TRAIN = (train_name,)
cfg.DATASETS.TEST = (test_name, )
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")

This is inference:
torch.backends.cudnn.benchmark = True
start = time.time()

predictor = DefaultPredictor(cfg)

im = cv2.imread("/content/drive/MyDrive/TEAM/mcocr_val_145114ixmyt.jpg")

outputs = predictor(im)

print(f"Inference time per image is : {(time.time() - start)} s")

Return time:
Inference time per image is : 2.7835421562194824 s
Image I inference size 1024 x 1024 pixel. I have change different size but it still inference 3 second / image. Am I missing anything about Detectron2 ?
More information GPU
enter image description here

Output:
There is a third way. You could use a faster toolkit for the inference e.g. OpenVINO. OpenVINO is optimized specifically for Intel hardware but it should work with any CPU. It optimizes your model by converting to Intermediate Represantation (IR), performing graph pruning and fusing some operations into others while preserving accuracy. Then it uses vectorization in runtime.
If you are able to export Detectron2 to ONNX model you can utilize OpenVINO. You can find a full tutorial on how to convert the ONNX model and performance comparison here. Some snippets below.
Install OpenVINO
The easiest way to do it is using PIP, especially when you use Google Colab.
pip install openvino-dev[onnx]

Use Model Optimizer to convert ONNX model
The Model Optimizer is a command line tool which comes from OpenVINO Development Package. It converts the ONNX model to IR, which is a default format for OpenVINO. You can also try the precision of FP16, which should give you better performance (just change data_type). Run in command line:
mo --input_model "model.onnx" --input_shape "[1,3, 224, 224]" --mean_values="[123.675, 116.28 , 103.53]" --scale_values="[58.395, 57.12 , 57.375]" --data_type FP32 --output_dir "model_ir"

Run the inference
The converted model can be loaded by the runtime and compiled for a specific device e.g. CPU or GPU (integrated into your CPU like Intel HD Graphics). If you don't know what is the best choice for you, just use AUTO.
# Load the network
ie = Core()
model_ir = ie.read_model(model="model_ir/model.xml")
compiled_model_ir = ie.compile_model(model=model_ir, device_name="CPU")

# Get output layer
output_layer_ir = compiled_model_ir.output(0)

# Run inference on the input image
result = compiled_model_ir([input_image])[output_layer_ir]

Disclaimer: I work on OpenVINO.


Instruction: Understanding custom policies in stable-baselines3

Input:
I was trying to understand the policy networks in stable-baselines3 from this doc page.

As explained in this example, to specify custom CNN feature extractor, we extend BaseFeaturesExtractor class and specify it in policy_kwarg.features_extractor_class with first param CnnPolicy:
model = PPO("CnnPolicy", "BreakoutNoFrameskip-v4", policy_kwargs=policy_kwargs)

Q1. Can we follow same approach for custom MLP feature extractor?

As explained in this example, to specify custom MLP feature extractor, we extend ActorCriticPolicy class and override _build_mlp_extractor() and pass it as first param:
class CustomActorCriticPolicy(ActorCriticPolicy): ...
model = PPO(CustomActorCriticPolicy, "CartPole-v1", verbose=1)

Q2. Can we follow same approach for custom CNN feature extractor?

I feel either we can have CNN extractor or MLP extractor. So it makes no sense to pass MlpPolicy as first param to model and then specify CNN feature extractor in policy_kwarg.features_extractor_class as in this example. This result in following policy (containing both features_extractor and mlp_extractor), which I feel is incorrect:
ActorCriticPolicy(
  (features_extractor): Net(
    (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
    (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))
    (fc3): Linear(in_features=384, out_features=512, bias=True)
  )
  (mlp_extractor): MlpExtractor(
    (shared_net): Sequential(
      (0): Linear(in_features=512, out_features=64, bias=True)
      (1): ReLU()
    )
    (policy_net): Sequential(
      (0): Linear(in_features=64, out_features=32, bias=True)
      (1): ReLU()
      (2): Linear(in_features=32, out_features=16, bias=True)
      (3): ReLU()
    )
    (value_net): Sequential(
      (0): Linear(in_features=64, out_features=32, bias=True)
      (1): ReLU()
      (2): Linear(in_features=32, out_features=16, bias=True)
      (3): ReLU()
    )
  )
  (action_net): Linear(in_features=16, out_features=7, bias=True)
  (value_net): Linear(in_features=16, out_features=1, bias=True)
)

Q3. Am I correct with this understanding? If yes, then is one of the MLP or CNN feature extractor ignored?

Output:
What I can say after i went through all the library code. CnnPolicy is differ to MlpPolicy only in implemented default BaseFeatureExtraction class. This make sense only in a case when you are not trying to create your custom BaseFeatureExtraction class.
Let me try to explain, we can see two types of policies:
MlpPolicy = ActorCriticPolicy
CnnPolicy = ActorCriticCnnPolicy

And we can see that class ActorCriticCnnPolicy(ActorCriticPolicy)is just based on ActorCriticPolicy and we can see following in parameters:
features_extractor_class: Type[BaseFeaturesExtractor] = NatureCNN

where NatureCNN is a simple implementation of BaseFeatureExtractor with CNN layers.
Now lets move to your questions!
Here is one thing, we always have following structure of model:
FeatureExtractor -> MlpExtractor -> Policy\Value nets
You can specify feature_exctractor_class and net_arch in policy_kwargs.
Default feature_extractor_class is based on your selection of CnnPolicy or MlpPolicy, but if you specify your own class then there is no difference. So, you can just use MlpPolicy.
If you specify net_arch, for example 'net_arch':[64, dict(pi=[32, 16], vf=[32, 16])] you will have 1 dense layer for mlp_extractor that connects output from feature_extractor.
To summary, my suggestion is to follow the example that you found in your Q3.:

Specify your own feature_extractor with any structure of this network either cnn or dense or whatever you want. The output of feature_extractor should be FC layer.
Specify net_acrh parameter in the following structure: [x1, x2, ..., dict(pi=[px1, px2, ..], vf=[vx1, vx2, ...])] where the first part is specified size of FC layers for mlp_extractor and the second part (inside dict) corresponds to sizes of policy and value networks.


Instruction: How to reshape multichannel image with a PyTorch encoder?

Input:
I have a tensor with dimensions [18, 512, 512], representing grayscale heatmaps for up to 18 specific objects on a 512x512 image. In order to generate a suitable representation of this image for my conditional GAN, I need to reshape this tensor into a [512, 4, 4] shape using an encoder. However, I can't understand how this transformation can be achieved, since given dimensions apppear too mismatched for direct linear or convolutional transformations.
class HeatmapEncoder(torch.nn.Module):
  def __init__(self):
    # source = 18x512x512
    # target = 512x4x4
    self.encoder = torch.nn.Sequential(
        nn.Linear(),
        nn.ReLU(),
        nn.Linear()
    )

    def forward(self, x):
      pass

It is possible to use `nn.Flatten()' with start_dim=0 here, but the result will be a simplified tensor that can't be used as an input by the linear layer.
The decoder part is not especially important right now, since I only need a low-dimensional representation of the heatmaps to condition my GAN and not to recreate those images.

Output:
You can try a couple of different approaches for your problem, like viewing it as a one big vector then slowly reducing it to your size, or permuting dimensions and applying different operations etc. Since there is no full code for testing it on the actual problem, I can't really say which will work better, but my first instinct says convolution based dimension reductions is quite suitable for this problem. First code, then talk:
class ReduceConv(torch.nn.Module):
  def __init__(self, nin, nout, activ=nn.ReLU):
    super(ReduceConv, self).__init__()

    # source = Batch x nin x H x W
    # target = Batch x nout x (H/2) x (W/2)

    self.conv = nn.Sequential(
        nn.Conv2d(
            nin, nout,
            kernel_size=3,
            stride=1,
            padding=1),
        nn.Conv2d(
            nout, nout,
            kernel_size=3,
            stride = 2,
            padding = 1),
        nn.BatchNorm2d(nout),
        activ()
    )

  def forward(self, x):
    return self.conv(x)

class HeatmapEncoder(torch.nn.Module):
  def __init__(self):
    super(HeatmapEncoder, self).__init__()

    # source = 18x512x512
    # target = 512x4x4
    self.encoder = torch.nn.Sequential(
        ReduceConv(18, 32),       # out->  32 256 256
        ReduceConv(32, 64),       # out->  64 128 128
        ReduceConv(64, 64),       # out->  64 64 64
        ReduceConv(64, 64),       # out->  64 32 32
        ReduceConv(64, 128),      # out-> 128 16 16
        ReduceConv(128, 256),     # out-> 256 8 8
        ReduceConv(256, 512)      # out-> 512 4 4
    )

  def forward(self, x):
    return self.encoder(x)

# 10 is batch size
inp = torch.rand(10, 18, 512, 512)
enc = HeatmapEncoder()
out = enc(inp)
print(inp.shape)      # torch.Size([10, 18, 512, 512])
print(out.shape)      # torch.Size([10, 512, 4, 4])

It is essentially just a stack of convolution layers. Note that, in each ReduceConv layer input dimensions are halved by using stride=2 convolutions. You don't technically need the first convolution in the ReduceConv layer, but it is deep learning era, more the merrier :) I've also added BatchNorm after each reduction along with an activation function. 18 is regarded as channels when inputted to first convolution. This way channels build up to 512 while the width and height is halved after each operation. This encoder model probably not the best or most efficient one but it should be good enough for your problem.


Instruction: AttributeError in torch_geometric.transforms

Input:
I have a problem that I cannot understand: even though a module ‘torch_geometric.transforms’ has an attribute ‘AddTrainValTestMask’ according to documentation , I cannot import it. I keep receiving an error AttributeError: module 'torch_geometric.transforms' has no attribute 'AddTrainValTestMask
My Pytorch version is 1.7.1
I took the code from here
Minimum reproducible example:
import os.path as osp

import torch
import torch.nn.functional as F
from torch_geometric.datasets import Planetoid
import torch_geometric.transforms as T
from torch_geometric.nn import SplineConv

dataset = 'Cora'
transform = T.Compose([
    T.AddTrainValTestMask('train_rest', num_val=500, num_test=500),
    T.TargetIndegree(),
])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=transform)
data = dataset[0]

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = SplineConv(dataset.num_features, 16, dim=1, kernel_size=2)
        self.conv2 = SplineConv(16, dataset.num_classes, dim=1, kernel_size=2)

    def forward(self):
        x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr
        x = F.dropout(x, training=self.training)
        x = F.elu(self.conv1(x, edge_index, edge_attr))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, edge_attr)
        return F.log_softmax(x, dim=1)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-3)

def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()

def test():
    model.eval()
    log_probs, accs = model(), []
    for _, mask in data('train_mask', 'test_mask'):
        pred = log_probs[mask].max(1)[1]
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs

for epoch in range(1, 201):
    train()
    log = 'Epoch: {:03d}, Train: {:.4f}, Test: {:.4f}'
    print(log.format(epoch, *test()))

Can anybody explain to me the problem?

Output: It has been renamed to RandomNodeSplit in the latest version of torch_geometric. You can directly use RandomNodeSplit to replace it.


Instruction: Worker timeout when preloading Pytorch model in Flask app on Render.com

Input:
In my app.py I have a function that uses a pretrained Pytorch model to generate keywords
@app.route('/get_keywords')
def get_keywords():
    generated_keywords = ml_controller.generate_keywords()
    return jsonify(keywords=generated_keywords)

and in ml_controller.py I have
def generate_keywords():
    model = load_keywords_model()
    output = model.generate()
    return output

This is working fine. Calls to /get_keywords correctly return the generated keywords. However this solution is quite slow since the model gets loaded on each call. Hence I tried to load the model just once by moving it outside my function:
model = load_keywords_model()

def generate_keywords():
    output = model.generate()
    return output

But now all calls to /get_keywords time out when I deploy my app to Render.com. (Locally it's working.) Strangely the problem is not that the model does not get loaded. When I write
model = load_keywords_model()
testOutput = model.generate()
print(testOutput)

def generate_keywords():
    output = model.generate()
    return output

a bunch of keywords are generated when I boot gunicorn. Also, all other endpoints that don't call ml_controller.generate_keywords() work without problems.
For testing purposes I also added a dummy function to ml_controller.py that I can call without problems
def dummy_string():
    return "dummy string"

Based on answers to similar problems I found, I'm starting Gunicorn with
gunicorn app:app --timeout 740 --preload --log-level debug

and in app.py I'm using
if __name__ == '__main__':
    app.run(debug=False, threaded=False)

However, the problem still persists.

Output:
The problem is that there's some bug that occurs for Pytorch models when Gunicorn is started with the --preload flag.
Render.com secretly adds this flag and doesn't show it in the settings which is why it took me days to figure this out. You can see all settings Render.com adds by calling printenv in the console.
To resolve the issue add a new environment variable
  GUNICORN_CMD_ARGS: '--access-logfile - --bind=0.0.0.0:10000'

which overwrites Render.com's standard settings
  GUNICORN_CMD_ARGS: '--preload --access-logfile - --bind=0.0.0.0:10000'


Instruction: Pytorch: load checkpoint from batch without iterating over dataset again

Input:
Instead of loading from an epoch wise checkpoint I need to be able to load from a batch. I am aware that this is not optimal but since I only have limited training time before my training gets interrupted (google colab free version) I need to be able to load from the batch it stopped or around that batch.
I also do not want to iterate over all data again but continue with the data the model has not seen yet.
My current approach which does not work:
def save_checkpoint(state, file=checkpoint_file):
  torch.save(state, file)

def load_checkpoint(checkpoint):
  model.load_state_dict(checkpoint['state_dict'])
  optimizer.load_state_dict(checkpoint['optimizer'])
  train_loss = checkpoint['train_loss']
  val_loss = checkpoint['val_loss']
  epoch = checkpoint['epoch']
  step = checkpoint['step']
  batch = checkpoint['batch']
  return model, optimizer, train_loss, val_loss, epoch, step, batch

While it does load the weights from where it stopped, it iterates over all data again.
Also, do I even need to capture train_loss and val_loss? I cannot see a difference in the loss being output when I include them or not. Thus, I assume it is already included in model.load_state_dict (?)
I assume capturing step and batch won't work this way and I actually need to include some sort of index tracker within my class DataSet ? I do already have this within the DataSet class
   def __getitem__(self, idx):
    question = self.data_qs[idx]
    answer1 = self.data_a1s[idx]
    answer2 = self.data_a2s[idx]
    target = self.targets[idx]

So, could this be useful?

Output:
You can achieve your goal by creating a custom Dataset class with a property self.start_index=step*batch and in your __getitem__ function the new index should be (self.start_index+idx)%len(self.data_qs)
If you create your Dataloader with shuffle=False then this tricks will work.
Additionally, With shuffle=True you can maintain a index mapper and needs to verify.


Instruction: Finding function/class definitions in PyTorch

Input:
I want to find out where certain classes and functions are defined within PyTorch (and other libraries).
Unfortunately, the following doesn't work:
import inspect
import torch

inspect.getsource(torch.tensor)

It throws the following error:
TypeError: module, class, method, function, traceback, frame, or code object was expected, got builtin_function_or_method

What's more, within PyCharm, I usually do 'gd' (in vim mode) to find a function/class definition, but this doesn't work either for PyTorch.
Please help me understand what is the problem here, and more importantly, how I can find these definitions in general.

Output:
this is actually complicated. Pytorch/libtorch is a huge project, and it relies on a lot of builtin low-level functions which have been implemented in C/Cuda. Most low-level kernels (math operations for example) even have several implementations, in order to optimize differently for the CPU and the GPU etc.
So there is a lot in this library that is not python code, and inspect is going to have a hard time.
If you want to find the source files, you are probably going to need to dive in the github repository yourself, and make good use of tools like grep and find.
However, the torch.nn module is almost entirely python, so I think inspect will work correctly on its features (like datasets, dataloaders, modules, optimizers etc)
And finally, if you need it, here is the file for torch.tensor, in which you will find python code mixed with calls to the C api : torch tensor source code
About the question in the comment below:
I cannot provide a full answer because this is reaching beyond my understanding of how exactly are python and C++ code interfaced in torch. But I'll do my best (If anyone has any correction or improvement to make to this, please do it).
There is a fundamental difference between C source code and python code : C/C++ are compiled and thus the features implemented in these languages are shipped as compiled assembly code. In other words, when your python code calls functions/objects from the underlying C code, it make calls to assembly functions that are not human-readable anymore. So the computer can make the calls, but the inspect feature that looks up for source code (for you to read) cannot work because this code does not exist anymore (at least not where inspect is looking). You would need other tools like a disassembler, a debugger etc, which are specialized in analyzing assembly instructions (you can also learn x86-64 assembly language programming if you are brave enough :D)