Instruction: How can I emulate/run PyTorch model that uses ATen STFT implementation on arm based cpu?

Input:
I am trying to run my PyTorch model for ASR on an arm based device without gpu. As far as I know, arm does not support MKL which ATen uses. Naturally, I am getting the following error when I try to make inference:
RuntimeError: fft: ATen not compiled with MKL support

How can I solve this problem? Are there any alternatives that I can use?

Output:
I solved this issue by bypassing PyTorch's stft implementation. This may not be feasible for everyone, but in my case it allowed me to make predictions using my model with no issues on arm device.
The problem stemmed from _VF.stft call in packages/torch/functional.py.
I changed the line
return _VF.stft(input, n_fft, hop_length, win_length, window, normalized, onesided, return_complex)

with:
librosa_stft = librosa.stft(input.cpu().detach().numpy().reshape(-1), n_fft, hop_length, win_length, window="hann", center=True, pad_mode=pad_mode)
librosa_stft = np.array([[a.real, a.imag] for a in librosa_stft])
librosa_stft = np.transpose(librosa_stft, axes=[0, 2, 1])
librosa_stft = np.expand_dims(librosa_stft, 0)
librosa_stft = torch.from_numpy(librosa_stft)
return librosa_stft

This code may be optimized further. I just tried to replicate what PyTorch did by using Librosa. Resulting output is same in both versions in my case. But you should check your outputs to be sure if you decide to use this method.


Instruction: while exporting T5 model to onnx using fastT5 getting "RuntimeError:output with shape [5, 8, 1, 2] doesn't match the broadcast shape [5, 8, 2, 2]"

Input:
i'm trying to convert T5 model to onnx using the fastT5 library, but
getting an error while running the following code
from fastT5 import export_and_get_onnx_model
from transformers import AutoTokenizer

model_name = 't5-small'
model = export_and_get_onnx_model(model_name)

tokenizer = AutoTokenizer.from_pretrained(model_name)
t_input = "translate English to French: The universe is a dark forest."
token = tokenizer(t_input, return_tensors='pt')

tokens = model.generate(input_ids=token['input_ids'],
               attention_mask=token['attention_mask'],
               num_beams=2)

output = tokenizer.decode(tokens.squeeze(), skip_special_tokens=True)
print(output)

the error:
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py:244: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_mask.shape[1] < attention_mask.shape[1]:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-16-80094b7c4f6f> in <module>()
      7                     input_names=decoder_input_names,
      8                     output_names=decoder_output_names,
----> 9                     dynamic_axes=dyn_axis_params,
     10                     )

24 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/t5/modeling_t5.py in forward(self, hidden_states, mask, key_value_states, position_bias, past_key_value, layer_head_mask, query_length, use_cache, output_attentions)
    497                 position_bias = position_bias + mask  # (batch_size, n_heads, seq_length, key_length)
    498
--> 499         scores += position_bias
    500         attn_weights = F.softmax(scores.float(), dim=-1).type_as(
    501             scores

RuntimeError: output with shape [5, 8, 1, 2] doesn't match the broadcast shape [5, 8, 2, 2]

can someone please help me solve the issue?

thank you.

Output:
I've checked the repository, it looks like a known issue as reported here : https://github.com/Ki6an/fastT5/issues/1
Developer of the library has posted a solution and created a notebook file here: https://colab.research.google.com/drive/1HuH1Ui3pCBS22hW4djIOyUBP5UW93705?usp=sharing
Solution is to modify modeling_t5.py file, at line 494 :
# Define this at line 426:
int_seq_length = int(seq_length)

# Change this at line 494:
position_bias = position_bias[:, :, -seq_length:, :]
position_bias = position_bias[:, :, -int_seq_length:, :]  # Updated version

If you don't want to modify the file yourself, you will need to wait until this pull request to be merged into Transformers library.


Instruction: I can iterate through my first iteration of PyTorch successfully, but am unable to do a second iteration. Please see code and error I get

Input:
I can iterate through my first iteration of PyTorch successfully, but am unable to do a second iteration.  Please see my code and error I get if I were to run my code a second time through.

current_grad = 0
l_rate = 10**-4
x=torch.tensor([[1.0,2.,4.],[2.,3.,2.]])
y=torch.tensor([1.0,0.])
w=torch.tensor([.5,2.,1.], requires_grad=True)

# forward propagate
output = x @ w.T
y_pred = 1/(1+ math.e**-(output))

# objective function
loss = sum(y*(y_pred**.5) + ((1-y) * (1-y_pred)**.5)) / len(y_pred)

# now, get gradient over these layers
x.requires_grad =True
y.requires_grad =True
w.requires_grad =True
loss.backward()

# update only 1 set of weights here.
with torch.no_grad():
    w = w + (w.grad * l_rate)

I get an error at my loss.backward line:
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
How can I fix this so that my w.grad is not NoneType on the second time around?

Output:
The main issue with your code is that w = w + (w.grad * l_rate) is assigning a new variable to w.
Instead you need to update w in place.
with torch.no_grad():
    w.copy_(w + w.grad * l_rate)

Some other issues

Learning rate is much too small for this problem
You need to zero the gradients of w after each step since backwards accumulates gradients.
Setting requires_grad for x and y is unnecessary since you don't need the gradient of loss w.r.t. these tensors.
In your code, the value of "loss" is actually something you want to maximize, since your objective function is maximal when y = y_pred. Generally we wouldn't call this loss, because that would imply you want to minimize it (only a masochist wants to maximize their loss :P).

Correcting for these issues
import torch

l_rate = 0.1
x = torch.tensor([[1.0, 2., 4.], [2., 3., 2.]])
y = torch.tensor([1.0, 0.])
w = torch.tensor([.5, 2., 1.], requires_grad=True)

# training loop ...

# forward propagate
output = x @ w
y_pred = torch.sigmoid(output)

# objective function
objective = torch.mean(y * (y_pred**.5) + ((1 - y) * (1 - y_pred)**.5))

# compute gradient of objective w.r.t. w
objective.backward()

with torch.no_grad():
    # gradient ascent
    w.copy_(w + (w.grad * l_rate))
    # zero the gradients
    w.grad.zero_()


Instruction: Transfer Learning Segmentation Model Perfoming Significantly Worse on Test Data

Input:
I am quite new to the field of semantic segmentation and have recently tried to run the code provided on this paper: Transfer Learning for Brain Tumor Segmentation that was made available on GitHub. It is a semantic segmentation task that uses the BraTS2020 dataset, comprising of 4 modalities, T1, T1ce, T2 and FLAIR. The author utilised a transfer learning approach using Resnet34 weights.
Due to hardware constraints, I had to half the batch size from 24 to 12. However, after training the model, I noticed a significant drop in performance, with the Dice Score (higher is better) of the 3 classes being only around 5-19-11 as opposed to the reported result of 78-87-82 in the paper. The training and validation accuracies however, seem to be performing normally, just that the model does not perform well on test data, I selected the model that was produced before overfitting (validation loss starts increasing but training loss still decreasing) but yielded equally bad results.
So far I have tried:

Decreasing the learning rate from 1e-3 to 1e-4, yielded similar results
Increased the number of batches fed to the model per training epoch to 200 batches per epoch, to match the number of iterations ran in the paper since I effectively halved the batch size - (100 batches per epoch, batch size of 24)

I noticed that image augmentations were applied to the training and validation dataset to increase the robustness of the model training. Do these augmentations need to be performed on the test set in order to make predictions? There are no resizing transforms, transforms that are present are Gaussian Blur and Noise, change in brightness intensity, rotations, elastic deformation, and mirroring, all implemented using the example here.
I'd greatly appreciate help on these questions:

By doubling the number of batches per epoch, it effectively matches the number of iterations performed as in the original paper since the batch size is halved. Is this the correct approach?

Does the test set data need to be augmented similarly to the training data in order to perform predictions? (Note: no resizing transformations were performed)

Output:
Technically, for a smaller batch the number of iterations should be higher for convergence. So, your approach is going to help, but it probably won't give the same performance boost as doubling the batch size.

Usually, we don't use augmentation on test data. But if the transformation applied on training and validation is not applied to the test data, the test performance will be poor, no doubt. You can try test time augmentation though, even though it's not very common for segmentation tasks https://github.com/qubvel/ttach


Instruction: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Input:
I am doing training and put the dataset inside the data folder. The Structure looks like this.
--data
-----mars
---------bbox_train
---------bbox_test
---------info

Many developers said that this is a label problem but I am not sure because labels are in the right place.
Traceback (most recent call last):
Args:Namespace(arch='resnet50graphpoolparthyper', concat=False, dataset='mars', dropout=0.1, eval_step=100, evaluate=False, gamma=0.1, gpu_devices='0', height=256, htri_only=False, lr=0.0003, margin=0.3, max_epoch=800, nheads=8, nhid=512, num_instances=4, part1=4, part2=8, part3=2, pool='avg', pretrained_model='/home/jiyang/Workspace/Works/video-person-reid/3dconv-person-reid/pretrained_models/resnet-50-kinetics.pth', print_freq=80, save_dir='log_hypergraphsagepart', seed=1, seq_len=8, start_epoch=0, stepsize=200, test_batch=1, train_batch=32, use_cpu=False, warmup=True, weight_decay=0.0005, width=128, workers=4, xent_only=False)
==========
Currently using GPU 0
Initializing dataset mars
=> MARS loaded
Dataset statistics:
  ------------------------------
  subset   | # ids | # tracklets
  ------------------------------
  train    |   625 |     8298
  query    |   626 |     1980
  gallery  |   622 |     9330
  ------------------------------
  total    |  1251 |    19608
  number of images per tracklet: 2 ~ 920, average 59.5
  ------------------------------
Initializing model: resnet50graphpoolparthyper
Model size: 44.17957M
==> Epoch 1/800  lr:1.785e-05
Traceback (most recent call last):
  File "main_video_person_reid_hypergraphsage_part.py", line 357, in <module>
    main()
  File "main_video_person_reid_hypergraphsage_part.py", line 220, in main
    train(model, criterion_xent, criterion_htri, optimizer, trainloader, use_gpu)
  File "main_video_person_reid_hypergraphsage_part.py", line 257, in train
    outputs, features = model(imgs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/models/ResNet_hypergraphsage_part.py", line 621, in forward
    x = self.base(x)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/models/resnet.py", line 213, in forward
    x = self.conv1(x)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Output:
Installing torch with CUDA 11.1 with the following command did fix the initial issue with torch 1.8:
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: Maximum Input Length of words/sentences of the Pegasus Model in the Transformers library

Input: In the Transformers library, what is the maximum input length of words and/or sentences of the Pegasus model? I read in the Pegasus research paper that the max was 512 tokens, but how many words and/or sentences is that? Also, can you increase the maximum number of 512 tokens?

Output:
In the Transformers library, what is the maximum input length of words and/or sentences of the Pegasus model?
It actually depends on your pretraining. You can create a pegagsus model that supports a length of 100 tokens or 10000 tokens. For example the model google/pegasus-cnn_dailymail supports 1024 tokens, while google/pegasus-xsum supports 512:

from transformers import PegasusTokenizerFast

t = PegasusTokenizerFast.from_pretrained("google/pegasus-xsum")
t2 = PegasusTokenizerFast.from_pretrained("google/pegasus-cnn_dailymail")
print(t.max_len_single_sentence)
print(t2.max_len_single_sentence)

Output:
511
1023

The numbers are reduced by one because of the special token  that is added to each sequence.

I read in the Pegasus research paper that the max was 512 tokens, but how many words and/or sentences is that?

That depends on your vocabulary.
from transformers import PegasusTokenizerFast
t = PegasusTokenizerFast.from_pretrained("google/pegasus-xsum")
print(t.tokenize('This is a test sentence'))
print("I know {} tokens".format(len(t)))

Output:
['▁This', '▁is', '▁a', '▁test', '▁sentence']
I know 96103 tokens

A word can be a token but it can also be split into several tokens:
print(t.tokenize('neuropsychiatric conditions'))

Output:
['▁neuro', 'psych', 'i', 'atric', '▁conditions']

Also, can you increase the maximum number of 512 tokens?

Yes, you can train a model with a pegasus architecture for a different input length but this is costly.


Instruction: How to use fine-tuned BERT model for sentence encoding?

Input:
I fine-tuned the BERT base model on my own dataset following the script here:
https://github.com/cedrickchee/pytorch-pretrained-BERT/tree/master/examples/lm_finetuning
I saved the model as a .pt file and I want to use it now for a sentence similarity task. Unfortunately, it is not clear to me, how to load the fine-tuned model. I tried the following:
model = BertModel.from_pretrained('trained_model.pt')
model.eval()

This doesn't work. It says:
ReadError: not a gzip file

So apparently, loading a .pt file with the from_pretrained method is not possible. Can anyone help me out here? Thank's a lot!! :)
Edit: I saved the model in a s3 bucket as follows:
# Convert model to buffer
buffer = io.BytesIO()
torch.save(model, buffer)
# Save in s3 bucket
output_model_file = output_folder + "trained_model.pt"
s3_.put_object(Bucket="power-plant-embeddings", Key=output_model_file, Body=buffer.getvalue())

Output:
To load a model with BertModel.from_pretrained() you need to have saved it using save_pretrained() (link).
Any other storage method would require the corresponding load. I am not familiar with S3, but I assume you can use get_object (link) to retrieve the model, and then save it using the huggingface api. From then on, you should be able to use from_pretrained() normally.


Instruction: My OpenCV Live Webcam Demo Doesn't Show Accurate Emotions

Input:
I developed a code by following a non-complete tutorial for emotion recognition with using PyTorch. I had so many errors but i fixed them thanks to the other questions here. But i simply stucked at this one. I am running my code, then my webcam opens but i see "Neutral" emotions all the time, even tough i perform different emotions. But on the console i am seeing other emotions as recorded, like:
tensor(0, device='cuda:0')
tensor(6, device='cuda:0')
tensor(4, device='cuda:0')
tensor(4, device='cuda:0')

You have any idea?
Here's my code:
import cv2 ### pip install opencv-python
##pip install open-cv-contrib-python fullpackage
#from deepface import DeepFace ##pip install deepface
import numpy as np

path = "haarcascade_frontalface_default.xml"
font_scale = 1.5
font = cv2.FONT_HERSHEY_PLAIN

#set the rectangle background to white
rectangle_bgr = (255, 255, 255)
# make a black image
img = np.zeros((500, 500))
#set some text
text = "VİDGA Projesi"
#get the width and height of the text box
(text_width, text_height) = cv2.getTextSize(text, font, fontScale=font_scale, thickness=1)[0]
#set the text start position
text_offset_x = 10
text_offset_y = img.shape[0] - 25
#make the coords of the box with a small padding of two pixels
box_coords = ((text_offset_x, text_offset_y), (text_offset_x + text_width + 2, text_offset_y - text_height -2))
cv2.rectangle(img, box_coords[0], box_coords[1], rectangle_bgr, cv2.FILLED)
cv2.putText(img, text, (text_offset_x, text_offset_y), font, fontScale=font_scale, color= (0,0,0), thickness=1)

cap = cv2.VideoCapture(1)
#Check if the webcam is opened correctly
if not cap.isOpened():
    cap = cv2.VideoCapture(0)
if not cap.isOpened():
    raise IOError("Cannot open webcam")

while True:
    ret, frame = cap.read()
    #eye_Cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_eye.xml')
    faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

    if ret == True:

        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        #print(faceCascade.empty())
        faces = faceCascade.detectMultiScale(gray,1.1,4)
        for x,y,w,h in faces:
            roi_gray = gray[y:y+h, x:x+w]
            roi_color = frame[y:y+h, x:x+w]
            cv2.rectangle(frame, (x,y), (x+w, y+h), (255, 0, 0), 2)
            facess = faceCascade.detectMultiScale(roi_gray)
            if len(facess) == 0:
                print("Face not detected")
            else:
                for(ex,ey,ew,eh) in facess:
                    face_roi = roi_color[ey: ey+eh, ex: ex+ew] ##cropping the face

    graytemp = cv2.cvtColor(face_roi, cv2.COLOR_BGR2GRAY)
    final_image = cv2.resize(graytemp, (48,48))
    final_image = np.expand_dims(final_image, axis =0) #add third dimension
    final_image = np.expand_dims(final_image, axis =0) #add fourth dimension
    final_image = final_image/255.0 # normalization
    dataa = torch.from_numpy(final_image)
    dataa = dataa.type(torch.FloatTensor)
    dataa = dataa.to(device)
    outputs = net(dataa)
    Pred = F.softmax(outputs, dim=1)
    Predictions = torch.argmax(Pred)
    print(Predictions)

    font = cv2.FONT_HERSHEY_SIMPLEX

    font_scale = 1.5
    font = cv2.FONT_HERSHEY_PLAIN

    if ((Predictions)==0):
        status = "Angry"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==1):
        status = "Disgust"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==2):
        status = "Fear"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==3):
        status = "Happy"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==4):
        status = "Sad"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==5):
        status = "Surprised"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

    elif ((Predictions)==6):
        status = "Neutral"

        x1,y1,w1,h1, = 0,0,175,5
        # Draw black background rectangle
        cv2.rectangle(frame, (x1, x1), (x1 + w1, y1 + h1), (0,0,0), -1)
        # Add text
        cv2.putText(frame, status, (x1 + int(w1/10), y1 + int(h1/2)), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 2)

        cv2.putText(frame, status, (100,150), font, 3,(0,0,255), 2, cv2.LINE_4)

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255))

        if ret == True:
            cv2.imshow('VIDGA Emotion Recognition', frame)
            if cv2.waitKey(2) & 0xFF == ord('q'):
                break

cap.release()
cv2.destroyAllWindows

Output:
Predictions is a tensor, so you need its value and not the tensor itself.
Change this line:
Predictions = torch.argmax(Pred)

with this:
Predictions = torch.argmax(Pred).item()


Instruction: RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. GPU not detected by pytorch

Input:
Having trouble with CUDA + Pytorch this is the error. I reinstalled CUDA and cudnn multiple times.
Conda env is detecting GPU but its giving errors with pytorch and certain cuda libraries. I tried with Cuda 10.1 and 10.0, and cudnn version 8 and 7.6.5, Added cuda to path and everything.
However anaconda is showing cuda tool kit 9.0 is installed, whilst I clearly installed 10.0, so I am not entirely sure what's the deal with that.
    => loading model from models/pytorch/pose_coco/pose_hrnet_w32_256x192.pth
Traceback (most recent call last):
  File "hydroman2.py", line 580, in <module>
    pose_model.load_state_dict(torch.load(cfg.TEST.MODEL_FILE), strict=False)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 773, in _legacy_load
    result = unpickler.load()
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 729, in persistent_load
    deserialized_objects[root_key] = restore_location(obj, location)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 178, in default_restore_location
    result = fn(storage, location)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 154, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 138, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

System info
   System info:
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2021-03-19 19:59:06.957967
UTC start time                                : 2021-03-19 15:59:06.957967
Running time (s)                              : 4.003899

__Hardware Information__
Machine                                       : AMD64
CPU Name                                      : znver1
CPU Count                                     : 12
Number of accessible CPUs                     : 12
List of accessible CPUs cores                 : 0 1 2 3 4 5 6 7 8 9 10 11
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  : 64bit adx aes avx avx2 bmi bmi2
                                                clflushopt clzero cmov cx16 cx8
                                                f16c fma fsgsbase fxsr lzcnt mmx
                                                movbe mwaitx pclmul popcnt prfchw
                                                rdrnd rdseed sahf sha sse sse2
                                                sse3 sse4.1 sse4.2 sse4a ssse3
                                                xsave xsavec xsaveopt xsaves

Memory Total (MB)                             : 16334
Memory Available (MB)                         : 8787

__OS Information__
Platform Name                                 : Windows-10-10.0.19041-SP0
Platform Release                              : 10
OS Name                                       : Windows
OS Version                                    : 10.0.19041
OS Specific Version                           : 10 10.0.19041 SP0 Multiprocessor Free
Libc Version                                  : ?

__Python Information__
Python Compiler                               : MSC v.1916 64 bit (AMD64)
Python Implementation                         : CPython
Python Version                                : 3.8.5
Python Locale                                 : en_US.cp1252

__LLVM Information__
LLVM Version                                  : 10.0.1

__CUDA Information__
CUDA Device Initialized                       : True
CUDA Driver Version                           : 11020
CUDA Detect Output:
Found 1 CUDA devices
id 0     b'GeForce GTX 1070'                              [SUPPORTED]
                      compute capability: 6.1
                           pci device id: 0
                              pci bus id: 6
Summary:
        1/1 devices are supported

CUDA Librairies Test Output:
Finding cublas from <unknown>
        named  cublas.dll
        trying to open library...       ERROR: failed to open cublas:
Could not find module 'cublas.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cusparse from <unknown>
        named  cusparse.dll
        trying to open library...       ERROR: failed to open cusparse:
Could not find module 'cusparse.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cufft from <unknown>
        named  cufft.dll
        trying to open library...       ERROR: failed to open cufft:
Could not find module 'cufft.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding curand from <unknown>
        named  curand.dll
        trying to open library...       ERROR: failed to open curand:
Could not find module 'curand.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding nvvm from <unknown>
        named  nvvm.dll
        trying to open library...       ERROR: failed to open nvvm:
Could not find module 'nvvm.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cudart from <unknown>
        named  cudart.dll
        trying to open library...       ERROR: failed to open cudart:
Could not find module 'cudart.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding libdevice from <unknown>
        searching for compute_20...     ERROR: can't open libdevice for compute_20
        searching for compute_30...     ERROR: can't open libdevice for compute_30
        searching for compute_35...     ERROR: can't open libdevice for compute_35
        searching for compute_50...     ERROR: can't open libdevice for compute_50

__ROC information__
ROC Available                                 : False
ROC Toolchains                                : None
HSA Agents Count                              : 0
HSA Agents:
None
HSA Discrete GPUs Count                       : 0
HSA Discrete GPUs                             : None

__SVML Information__
SVML State, config.USING_SVML                 : True
SVML Library Loaded                           : True
llvmlite Using SVML Patched LLVM              : True
SVML Operational                              : True

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : True
+-->Vendor: MS
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

__Numba Environment Variable Information__
None found.

__Conda Information__
Conda Build                                   : 3.20.5
Conda Env                                     : 4.9.2
Conda Platform                                : win-64
Conda Python Version                          : 3.8.5.final.0
Conda Root Writable                           : True

__Installed Packages__
_pytorch_select           1.1.0                       cpu    anaconda
_tflow_select             2.3.0                       mkl    anaconda
absl-py                   0.12.0                   pypi_0    pypi
alabaster                 0.7.12                   pypi_0    pypi
appdirs                   1.4.3            py36h28b3542_0    anaconda
argparse                  1.4.0                    pypi_0    pypi
asn1crypto                1.3.0                    py36_0    anaconda
astor                     0.8.1              pyh9f0ad1d_0    conda-forge
astunparse                1.6.3                    pypi_0    pypi
atomicwrites              1.4.0                      py_0    anaconda
attrs                     19.3.0                     py_0    anaconda
babel                     2.9.0                    pypi_0    pypi
backcall                  0.2.0                      py_0    anaconda
backports                 1.0                        py_2    anaconda
backports.weakref         1.0.post1       py36h9f0ad1d_1001    conda-forge
blas                      1.0                         mkl    anaconda
bleach                    1.5.0                    py36_0    conda-forge
blinker                   1.4                        py_1    conda-forge
brotlipy                  0.7.0           py36he774522_1000    anaconda
bzip2                     1.0.8                he774522_0    anaconda
ca-certificates           2020.10.14                    0    anaconda
cachetools                4.1.1                      py_0    anaconda
certifi                   2020.6.20                py36_0    anaconda
cffi                      1.14.0           py36h7a1dbc1_0    anaconda
chardet                   3.0.4                 py36_1003    anaconda
click                     7.1.2              pyh9f0ad1d_0    conda-forge
cloudpickle               1.4.1                      py_0    anaconda
colorama                  0.4.3                      py_0    anaconda
contextlib2               0.6.0.post1                py_0    anaconda
cpuonly                   1.0                           0    pytorch
cryptography              2.9.2            py36h7a1dbc1_0    anaconda
cudatoolkit               9.0                           1    anaconda
cudnn                     7.6.5                 cuda9.0_0    anaconda
curl                      7.71.0               h2a8f88b_0    anaconda
cycler                    0.10.0           py36h009560c_0    anaconda
cython                    0.29.22                  pypi_0    pypi
cytoolz                   0.10.1           py36he774522_0    anaconda
dask-core                 2.19.0                     py_0    anaconda
decorator                 4.4.2                      py_0    anaconda
defusedxml                0.6.0                      py_0    anaconda
dlib                      19.20            py36h5653133_1    conda-forge
docker-py                 4.2.1            py36h9f0ad1d_0    conda-forge
docker-pycreds            0.4.0                      py_0    anaconda
docutils                  0.16                     pypi_0    pypi
easydict                  1.7                      pypi_0    pypi
entrypoints               0.3                      py36_0    anaconda
ffmpeg                    2.7.0                         0    menpo
flake8                    3.8.3                      py_0    anaconda
flake8-polyfill           1.0.2                    py36_0    anaconda
flake8-quotes             3.0.0              pyh9f0ad1d_0    conda-forge
flatbuffers               1.12                     pypi_0    pypi
freetype                  2.10.2               hd328e21_0    anaconda
gast                      0.2.2                    pypi_0    pypi
geos                      3.8.1                h33f27b4_0    anaconda
gettext                   0.19.8.1          hb01d8f6_1002    conda-forge
git                       2.23.0               h6bb4b03_0    anaconda
glib                      2.58.3          py36h04c7ab9_1004    conda-forge
google-auth               1.28.0                   pypi_0    pypi
google-auth-oauthlib      0.4.3                    pypi_0    pypi
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
grpcio                    1.32.0                   pypi_0    pypi
h5py                      2.10.0           py36h5e291fa_0    anaconda
hdf5                      1.10.4               h7ebc959_0    anaconda
html5lib                  0.9999999                py36_0    conda-forge
icc_rt                    2019.0.0             h0cc432a_1    anaconda
icu                       58.2                 ha925a31_3    anaconda
idna                      2.10                       py_0    anaconda
imageio                   2.8.0                      py_0    anaconda
imageio-ffmpeg            0.4.2                      py_0    conda-forge
imagesize                 1.2.0                    pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
importlib-metadata        1.7.0                    py36_0    anaconda
importlib_metadata        1.7.0                         0    anaconda
intel-openmp              2019.4                      245    anaconda
ipykernel                 5.3.0            py36h5ca1d4c_0    anaconda
ipyparallel               6.3.0                    pypi_0    pypi
ipython                   7.16.1           py36h5ca1d4c_0    anaconda
ipython_genutils          0.2.0                    py36_0    anaconda
ipywidgets                7.5.1                      py_0    anaconda
jedi                      0.17.1                   py36_0    anaconda
jinja2                    2.11.2                     py_0    anaconda
joblib                    0.15.1                     py_0    anaconda
jpeg                      9d                   he774522_0    conda-forge
json-tricks               3.15.5                   pypi_0    pypi
jsonschema                3.2.0                    py36_0    anaconda
jupyter                   1.0.0                    py36_7    anaconda
jupyter_client            6.1.3                      py_0    anaconda
jupyter_console           6.1.0                      py_0    anaconda
jupyter_core              4.6.3                    py36_0    anaconda
keras-applications        1.0.8                      py_1    anaconda
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.2.0            py36h74a9793_0    anaconda
krb5                      1.18.2               hc04afaa_0    anaconda
leptonica                 1.78.0               h919f142_2    conda-forge
libarchive                3.3.3                h0643e63_5    anaconda
libcurl                   7.71.0               h2a8f88b_0    anaconda
libffi                    3.2.1             h6538335_1007    conda-forge
libgpuarray               0.7.6             hfa6e2cd_1003    conda-forge
libiconv                  1.15             vc14h29686d3_5  [vc14]  anaconda
libmklml                  2019.0.5                      0    anaconda
libpng                    1.6.37               h2a8f88b_0    anaconda
libprotobuf               3.12.3               h7bd577a_0    anaconda
libsodium                 1.0.18               h62dcd97_0    anaconda
libssh2                   1.9.0                h7a1dbc1_1    anaconda
libtiff                   4.1.0                h56a325e_0    anaconda
libwebp                   1.0.2                hfa6e2cd_5    conda-forge
libxml2                   2.9.10               h464c3ec_1    anaconda
libxslt                   1.1.34               he774522_0    anaconda
lxml                      4.5.0            py36h1350720_0    anaconda
lz4-c                     1.8.1.2              h2fa13f4_0    anaconda
lzo                       2.10                 he774522_2    anaconda
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
mako                      1.1.0                      py_0    anaconda
markdown                  3.3.4                    pypi_0    pypi
markupsafe                1.1.1            py36he774522_0    anaconda
matplotlib                3.1.3                    py36_0    anaconda
matplotlib-base           3.1.3            py36h64f37c6_0    anaconda
mccabe                    0.6.1                    py36_1    anaconda
mistune                   0.8.4            py36he774522_0    anaconda
mkl                       2018.0.3                      1    anaconda
mkl_fft                   1.0.6            py36hdbbee80_0    anaconda
mkl_random                1.0.1            py36h77b88f5_1    anaconda
mock                      4.0.3                    pypi_0    pypi
more-itertools            8.4.0                      py_0    anaconda
moviepy                   1.0.1                      py_0    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
nbconvert                 5.6.1                    py36_0    anaconda
nbformat                  5.0.7                      py_0    anaconda
networkx                  2.4                        py_0    anaconda
ninja                     1.9.0            py36h74a9793_0    anaconda
nose                      1.3.7                    pypi_0    pypi
notebook                  6.0.3                    py36_0    anaconda
numpy                     1.19.5                   pypi_0    pypi
oauthlib                  3.1.0                      py_0    anaconda
olefile                   0.46                     py36_0    anaconda
opencv-python             3.4.1.15                 pypi_0    pypi
openjpeg                  2.3.1                h57dd2e7_3    conda-forge
openssl                   1.1.1h               he774522_0    anaconda
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 20.4                       py_0    anaconda
pandas                    1.0.3            py36h47e9c7a_0    anaconda
pandoc                    2.9.2.1                       0    anaconda
pandocfilters             1.4.2                    py36_1    anaconda
parso                     0.7.0                      py_0    anaconda
pcre                      8.44                 ha925a31_0    anaconda
pep8-naming               0.8.2                    py36_0    anaconda
pickleshare               0.7.5                    py36_0    anaconda
pillow                    7.1.2            py36hcc1f983_0    anaconda
pip                       20.2.4                   py36_0    anaconda
pluggy                    0.13.1                   py36_0    anaconda
poppler                   0.87.0               hdbe765f_0    conda-forge
poppler-data              0.4.9                         1    conda-forge
proglog                   0.1.9                      py_0    conda-forge
prometheus_client         0.8.0                      py_0    anaconda
prompt-toolkit            3.0.5                      py_0    anaconda
prompt_toolkit            3.0.5                         0    anaconda
protobuf                  3.12.3           py36h33f27b4_0    anaconda
psutil                    5.8.0                    pypi_0    pypi
py                        1.9.0                      py_0    anaconda
pyasn1                    0.4.8                      py_0    anaconda
pyasn1-modules            0.2.8                    pypi_0    pypi
pycocotools               2.0                      pypi_0    pypi
pycodestyle               2.6.0                      py_0    anaconda
pycparser                 2.20                       py_0    anaconda
pyflakes                  2.2.0                      py_0    anaconda
pygments                  2.6.1                      py_0    anaconda
pygpu                     0.7.6           py36h7725771_1001    conda-forge
pyjwt                     1.7.1                      py_0    conda-forge
pyopenssl                 19.1.0                   py36_0    anaconda
pyparsing                 2.4.7                      py_0    anaconda
pyqt                      5.9.2            py36h6538335_2    anaconda
pyreadline                2.1                   py36_1001    conda-forge
pyrsistent                0.16.0           py36he774522_0    anaconda
pysocks                   1.7.1                    py36_0    anaconda
pytesseract               0.3.3              pyh8c360ce_0    conda-forge
pytest                    5.4.3                    py36_0    anaconda
python                    3.6.10               h9f7ef89_1    anaconda
python-dateutil           2.8.1                      py_0    anaconda
python_abi                3.6                     1_cp36m    conda-forge
pytorch                   1.5.1               py3.6_cpu_0  [cpuonly]  pytorch
pytz                      2020.1                     py_0    anaconda
pywavelets                1.1.1            py36he774522_0    anaconda
pywin32                   223              py36hfa6e2cd_1    anaconda
pywinpty                  0.5.7                    py36_0    anaconda
pyyaml                    5.3.1            py36he774522_0    anaconda
pyzmq                     19.0.1           py36ha925a31_1    anaconda
qt                        5.9.7            vc14h73c81de_0  [vc14]  anaconda
qtconsole                 4.7.5                      py_0    anaconda
qtpy                      1.9.0                      py_0    anaconda
requests                  2.24.0                     py_0    anaconda
requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge
rsa                       4.6                pyh9f0ad1d_0    conda-forge
scikit-image              0.16.2           py36h47e9c7a_0    anaconda
scikit-learn              0.20.1           py36hb854c30_0    anaconda
scipy                     1.4.1                    pypi_0    pypi
send2trash                1.5.0                    py36_0    anaconda
setuptools                50.3.0           py36h9490d1a_1    anaconda
shapely                   1.6.4                    pypi_0    pypi
simplejson                3.17.0           py36he774522_0    anaconda
sip                       4.19.8           py36h6538335_0    anaconda
six                       1.15.0                     py_0    anaconda
sklearn                   0.0                      pypi_0    pypi
slidingwindow             0.0.14                   pypi_0    pypi
snowballstemmer           2.1.0                    pypi_0    pypi
sphinx                    3.5.2                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    1.0.3                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.4                    pypi_0    pypi
sqlite                    3.32.3               h2a8f88b_0    anaconda
swig                      3.0.12               h047fa9f_3    anaconda
tbb                       2020.0               h74a9793_0    anaconda
tbb4py                    2020.0           py36h74a9793_0    anaconda
tensorboard               1.13.1                   pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorboardx              1.6                        py_0    conda-forge
tensorflow                2.4.1                    pypi_0    pypi
tensorflow-estimator      1.13.0                   pypi_0    pypi
tensorflow-gpu            1.13.1                   pypi_0    pypi
tensorflow-gpu-estimator  2.1.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.8.3                    py36_0    anaconda
testpath                  0.4.4                      py_0    anaconda
theano                    1.0.4           py36h003fed8_1002    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    anaconda
tk                        8.6.10               he774522_0    anaconda
toolz                     0.10.0                     py_0    anaconda
torchfile                 0.1.0                      py_0    conda-forge
torchvision               0.6.1                  py36_cpu  [cpuonly]  pytorch
tornado                   6.0.4            py36he774522_1    anaconda
tqdm                      4.47.0                     py_0    anaconda
traitlets                 4.3.3                    py36_0    anaconda
typing-extensions         3.7.4.3                  pypi_0    pypi
urllib3                   1.25.11                    py_0    anaconda
vc                        14.1                 h0510ff6_4    anaconda
visdom                    0.1.8.9                       0    conda-forge
vs2015_runtime            14.16.27012          hf0eaf9b_3    anaconda
vs2017_win-64             19.16.27038          h2e3bad8_2    conda-forge
vswhere                   2.7.1                h21ff451_0    anaconda
wcwidth                   0.2.5                      py_0    anaconda
webencodings              0.5.1                    py36_1    anaconda
websocket-client          0.57.0                   py36_1    anaconda
werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge
wget                      1.16.3                        0    menpo
wheel                     0.35.1                     py_0    anaconda
widgetsnbextension        3.5.1                    py36_0    anaconda
win_inet_pton             1.1.0                    py36_0    anaconda
wincertstore              0.2              py36h7fe50ca_0    anaconda
winpty                    0.4.3                         4    anaconda
wrapt                     1.12.1           py36h68a101e_1    conda-forge
xz                        5.2.5                h62dcd97_0    anaconda
yacs                      0.1.8                    pypi_0    pypi
yaml                      0.1.7                hc54c509_2    anaconda
zeromq                    4.3.2                ha925a31_2    anaconda
zipp                      3.3.1                      py_0    anaconda
zlib                      1.2.11               h62dcd97_4    anaconda
zstd                      1.3.7                h508b16e_0    anaconda

No errors reported.

Output:
Solved.
Pytorch was installing CPU only version for some reason, reinstalling pytorch didn't help.
Uninstalling pytorch: conda uninstall pytorch
Followed by uninstalling cpu only: conda uninstall cpuonly
Then installing pytorch again solved it.


Instruction: Batch size and Training time

Input:
Thank you for @Prune's critical comments on my questions.
I am trying to find the relationship between batch size and training time by using MNIST dataset.
By reading numerous questions in stackoverflow, such as this one:
How does batch size impact time execution in neural networks?
people said that the training time will be decreased when I use small batch size.
However, by trying out these two, I found that training with batch size == 1 takes way more time than batch size == 60,000. I set epoch as 10.
I split my MMIST dataset into 60k for the training and 10k for the testing.
This below is my code and results.
mnist_trainset = torchvision.datasets.MNIST(root=root_dir, train=True,
                                download=True,
                                transform=transforms.Compose([transforms.ToTensor()]))

mnist_testset  = torchvision.datasets.MNIST(root=root_dir,
                                train=False,
                                download=True,
                                transform=transforms.Compose([transforms.ToTensor()]))

train_dataloader = torch.utils.data.DataLoader(mnist_trainset,
                                               batch_size=1,
                                               shuffle=True)

test_dataloader  = torch.utils.data.DataLoader(mnist_testset,
                                               batch_size=50,
                                               shuffle=False)

# Define the model
class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear_1 = torch.nn.Linear(784, 256)
        self.linear_2 = torch.nn.Linear(256, 10)
        self.sigmoid  = torch.nn.Sigmoid()

    def forward(self, x):
        x = x.reshape(x.size(0), -1)
        x = self.linear_1(x)
        x = self.sigmoid(x)
        pred = self.linear_2(x)

        return pred

# trainer
no_epochs = 10

def my_trainer(optimizer, model):

    criterion = torch.nn.CrossEntropyLoss()

    train_loss = list()
    test_loss  = list()
    test_acc   = list()
    best_test_loss = 1

    for epoch in range(no_epochs):

        # timer starts
        start = timer()

        total_train_loss = 0
        total_test_loss = 0

        # training
        # set up training mode
        model.train()

        for itr, (image, label) in enumerate(train_dataloader):

            optimizer.zero_grad()

            pred = model(image)

            loss = criterion(pred, label)
            total_train_loss += loss.item()

            loss.backward()
            optimizer.step()

        total_train_loss = total_train_loss / (itr + 1)
        train_loss.append(total_train_loss)

        # testing
        # change to evaluation mode
        model.eval()
        total = 0
        for itr, (image, label) in enumerate(test_dataloader):

            pred = model(image)

            loss = criterion(pred, label)
            total_test_loss += loss.item()

            # we now need softmax because we are testing.
            pred = torch.nn.functional.softmax(pred, dim=1)
            for i, p in enumerate(pred):
                if label[i] == torch.max(p.data, 0)[1]:
                    total = total + 1

        # caculate accuracy
        accuracy = total / len(mnist_testset)

        # append accuracy here
        test_acc.append(accuracy)

        # append test loss here
        total_test_loss = total_test_loss / (itr + 1)
        test_loss.append(total_test_loss)

        print('\nEpoch: {}/{}, Train Loss: {:.8f}, Test Loss: {:.8f}, Test Accuracy: {:.8f}'.format(epoch + 1, no_epochs, total_train_loss, total_test_loss, accuracy))

        if total_test_loss < best_test_loss:
            best_test_loss = total_test_loss
            print("Saving the model state dictionary for Epoch: {} with Test loss: {:.8f}".format(epoch + 1, total_test_loss))
            torch.save(model.state_dict(), "model.dth")

        # timer finishes
        end = timer()
        print(end - start)

    return no_epochs, test_acc, test_loss

model_sgd = Model()
optimizer_SGD = torch.optim.SGD(model_sgd.parameters(), lr=0.1)
sgd_no_epochs,      sgd_test_acc, sgd_test_loss           = my_trainer(optimizer=optimizer_SGD, model=model_sgd)

I calculated how much time did it took for each epoch.
And this below is the result.
Epoch: 1/10, Train Loss: 0.23193890, Test Loss: 0.12670580, Test Accuracy: 0.96230000
63.98903721500005 seconds

Epoch: 2/10, Train Loss: 0.10275097, Test Loss: 0.10111042, Test Accuracy: 0.96730000
63.97179028100004 seconds

Epoch: 3/10, Train Loss: 0.07269370, Test Loss: 0.09668248, Test Accuracy: 0.97150000
63.969843954 seconds

Epoch: 4/10, Train Loss: 0.05658571, Test Loss: 0.09841745, Test Accuracy: 0.97070000
64.24135530400008 seconds

Epoch: 5/10, Train Loss: 0.04183391, Test Loss: 0.09828428, Test Accuracy: 0.97230000
64.19695308500013 seconds

Epoch: 6/10, Train Loss: 0.03393899, Test Loss: 0.08982467, Test Accuracy: 0.97530000
63.96944059600014 seconds

Epoch: 7/10, Train Loss: 0.02808819, Test Loss: 0.08597597, Test Accuracy: 0.97700000
63.59837343000004 seconds

Epoch: 8/10, Train Loss: 0.01859330, Test Loss: 0.07529452, Test Accuracy: 0.97950000
63.591578820999985 seconds

Epoch: 9/10, Train Loss: 0.01383720, Test Loss: 0.08568452, Test Accuracy: 0.97820000
63.66664020100029

Epoch: 10/10, Train Loss: 0.00911216, Test Loss: 0.07377760, Test Accuracy: 0.98060000
63.92636473799985 seconds

After this I changed the batch size to 60000 and run the same program again.
train_dataloader = torch.utils.data.DataLoader(mnist_trainset,
                                               batch_size=60000,
                                               shuffle=True)

test_dataloader  = torch.utils.data.DataLoader(mnist_testset,
                                               batch_size=50,
                                               shuffle=False)

print("\n===== Entering SGD optimizer =====\n")
model_sgd = Model()
optimizer_SGD = torch.optim.SGD(model_sgd.parameters(), lr=0.1)
sgd_no_epochs,      sgd_test_acc, sgd_test_loss           = my_trainer(optimizer=optimizer_SGD, model=model_sgd)

I got this result for batch size == 60000
Epoch: 1/10, Train Loss: 2.32325006, Test Loss: 2.30074144, Test Accuracy: 0.11740000
6.54154992299982 seconds

Epoch: 2/10, Train Loss: 2.30010080, Test Loss: 2.29524792, Test Accuracy: 0.11790000
6.341824101999919 seconds

Epoch: 3/10, Train Loss: 2.29514933, Test Loss: 2.29183527, Test Accuracy: 0.11410000
6.161918789000083 seconds

Epoch: 4/10, Train Loss: 2.29196787, Test Loss: 2.28874513, Test Accuracy: 0.11450000
6.180891567999879 seconds

Epoch: 5/10, Train Loss: 2.28899717, Test Loss: 2.28571669, Test Accuracy: 0.11570000
6.1449509030003355 seconds

Epoch: 6/10, Train Loss: 2.28604794, Test Loss: 2.28270152, Test Accuracy: 0.11780000
6.311743144000047 seconds

Epoch: 7/10, Train Loss: 2.28307867, Test Loss: 2.27968731, Test Accuracy: 0.12250000
6.060618773999977 seconds

Epoch: 8/10, Train Loss: 2.28014660, Test Loss: 2.27666961, Test Accuracy: 0.12890000
6.171511712999745 seconds

Epoch: 9/10, Train Loss: 2.27718973, Test Loss: 2.27364607, Test Accuracy: 0.13930000
6.164125173999764 seconds

Epoch: 10/10, Train Loss: 2.27423453, Test Loss: 2.27061504, Test Accuracy: 0.15350000
6.077817454000069 seconds

As you can see it is clear that it took more time for each epoch when batch_size == 1 which is different from what I have seen.
Maybe I am confused with the training time per epoch vs the training time until convergence? Seems like my intuition is correct by looking at this webpage: https://medium.com/deep-learning-experiments/effect-of-batch-size-on-neural-net-training-c5ae8516e57
Can someone please explain what is happening?

Output:
This is a borderline question; you should still be able to extract this understanding from the basic literature ... eventually.
Your insight is exactly correct: you are measuring execution time per epoch, rather than total Time-to-Train (TTT).  You have also carried the generic "smaller batches" advice ad absurdum: a batch size of 1 is almost guaranteed to be sub-optimal.
The mechanics are very simple at a macro level.
With a batch size of 60k (the entire training set), you run all 60k images through the model, average their results, and then do one back-propagation for that average result.  This tends to lose the learning you can get from focusing on little-seen features.
With a batch size of 1, you run each image individually through the model, average the one result (a very simple operation :-) ), and do a back propagation.  This tends to over-emphasize individual effects, especially retaining superstitious effects from each single image.  It also gives too much weight to the initial assumptions of the first few images.
The most obvious effect of the tiny batch size is that you're doing 60k back-props instead of 1, so each epoch takes much longer.

Either of these approaches is an extreme case, usually absurd in application.
You need to experiment to find the "sweet spot" that gives you the fastest convergence to acceptable (near-optimal) accuracy.  There are a few considerations in choosing your experimental design:

Memory size: you want to be able to ingest the entire batch into memory at once.  This allows your model to pipeline reading and processing.  If you exceed available memory, you will lose a lot of time to swapping.  If you under-use the memory, you leave some potential performance untapped.
Processors: if you're on a multi-processor chip, you want to keep them all busy.  If you care to assign processors through your OS controls, you'll also want to play with how many to assign to model computation, and how many to assign to I/O and system use.  For instance, in one project I did, our group found that our 32 cores were best used with 28 allocated to computation, 4 reserved for I/O and other system functions.
Scaling: some characteristics work best in powers of 2.  You may find that a batch size that is 2^n or 3 * 2^n for some n, works best, simply because of block sizes and other system allocations.

The experimental design that has worked best for me over the years is to start with a power of 2 that is roughly the square root of the training set size.  For you, there's an obvious starting guess of 256.  Thus, you'd run experiments at perhaps 64, 128, 256, 512, and 1024.  See which ones give you the fastest convergence.
Then do one step of refinement, using that factor of 3.  For instance, if you find that the best performance comes at 128, also try 96 and 192.
You will likely see very little difference between your "sweet spot" and the adjacent batch sizes; this is the nature of most complex information systems.


Instruction: Spliting the dataset using SubsetRandomSampler not working

Input:
I used SubsetRandomSampler to split the training data to train (80%) and validation data (20%). But it is showing the same number of images for both after the split (4996):
>>> print('len(train_data): ', len(train_loader.dataset))
>>> print('len(valid_data): ', len(validation_loader.dataset))

len(train_data):  4996
len(valid_data):  4996

Full code:
import numpy as np
import torch
from torchvision import transforms
from torch.utils.data.sampler import SubsetRandomSampler

train_transforms = transforms.Compose([transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406],
                                                            [0.229, 0.224, 0.225])])
dataset = datasets.ImageFolder( '/data/images/train', transform=train_transforms )

validation_split = .2
shuffle_dataset = True
random_seed= 42
batch_size = 20

dataset_size = len(dataset) #4996
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))

if shuffle_dataset :
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, val_indices = indices[split:], indices[:split]

train_sampler = SubsetRandomSampler(train_indices)
valid_sampler = SubsetRandomSampler(val_indices)

train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler)
validation_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=valid_sampler)

Output:
train_loader.dataset and validation_loader.dataset are methods which return the underlying original dataset that the loaders sample from (i.e. the original dataset of size 4996).
If you iterate through the loaders themselves you will see they only return as many samples (acounting for batching) as you have included in the index for each sampler, however.


Instruction: Pytorch 1D tensors expected but got 2D tensors

Input:
I have been working on making neural network from scratch in python. The input tensor is of shape [400,3] and target_tensor has the shape [400]. I am getting error during taking derivative of weights. Below are functions
    def sigmoid(z):
           return 1 / (1 + torch.exp(-z))

    def nueral_net(data,weights,bias):
           return sigmoid( ( data @ weights ) + bias )

    def loss_function(prediction,actual,m):
           return (-1/m) * (torch.sum(actual * torch.log(prediction) + (1-actual)
           * torch.log(1- prediction)))

    w = torch.randn(input_tensor.shape[1],1)
    b = torch.randn(1,1)

    predictions = nueral_net(input_tensor.float() , w, b) #Applying model
    loss = loss_function(predictions,target_tensor.unsqueeze(1),400)
    dw = (1/400) * torch.dot(input_tensor,(predictions - target_tensor).T)

Running this throws an error.
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-26-632338d8fd16> in <module>
          1 predictions = nueral_net(input_tensor.float() , w, b) #Applying model
          2 loss = loss_function(predictions,target_tensor.unsqueeze(1),400)
    ----> 3 dw = (1/400) * torch.dot(input_tensor,(predictions - target_tensor).T)
          4 db = (1/400) * torch.sum(predictions - target_tensor)
          5 #m = input_tensor.shape[0]

    RuntimeError: 1D tensors expected, but got 2D and 2D tensor

Output:
If we see the doc of torch.dot :
torch.dot(input, other, *, out=None) → Tensor
Computes the dot product of two 1D tensors.
NOTE : Unlike NumPy’s dot, torch.dot intentionally only supports computing the dot product of two 1D tensors with the same number of elements.
Parameters
input (Tensor) – first tensor in the dot product, must be 1D.
other (Tensor) – second tensor in the dot product, must be 1D.
Coming to your question....both input_tensor & (predictions - target_tensor).T in 2D.
Please make it 1D


Instruction: Loading PyTorch model in multiprocessing Process flashes cmd

Input:
Problem:
When I open pytorch model (read as load state_dict from disk) in subprocess it pops up cmd window for couple ms which causes other programs to loose focus - annoying when working on something else etc.
I have traced the cause to 2 lines, both causing it in some circumstances and managed to reproduce it for one (the second one is when doing model.to(device))
main.py
model_path = 'testing\\agent\\model_test.pth'

# create model
from testing.agent.torch_model import Net
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # I have cuda available
m = Net()
m.to(device)
# save it
torch.save(m.state_dict(), model_path)

# open it in subprocess
from testing.agent.AgentOpenSim_Process import Open_Agent_Sim
p = Open_Agent_Sim(p=model_path, msgLogger=None)
p.start()

torch_model.py
(source pydocs: https://pytorch.org/tutorials/recipes/recipes/save_load_across_devices.html)
import torch.nn as nn
import torch.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

AgentOpenSim_Process.py
from multiprocessing import Queue, Process
import os, time, torch

from testing.agent.torch_model import Net

class Open_Agent_Sim(Process):
    def __init__(self, p:str, **kwargs):
        super(Process, self).__init__(daemon=True)
        self.path = p
        self._msgLogger = kwargs['msgLogger'] if kwargs['msgLogger'] is not None else Queue()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.device_cpu = torch.device("cpu")

    def __print(self, msg: str, verbosity):
        # personal tool for debugging multiprocessing (it sends messages to Queue and main process
        # reads them inside thread and prints into console... and yes i know about multiprocessing
        # logger, its used aswell)
        self._msgLogger.put(('Open_Agent_Sim: '+msg, verbosity))

    def run(self):
        self.__pid = os.getpid()
        try:
            self.__print('opening model',0)
            self.init_agent()
            self.__print('opening model - done',0)
        except Exception as e:
            # solved by custom exception wrapper
            pass
        else:
            self.__print('has ended',0)
            return

    def init_agent(self):
        # init instance
        self.__print('0a', 0)
        m = Net()
        self.__print('0b', 0)
        time.sleep(2)
        # load state dict
        self.__print('1a', 0)
        l = torch.load(self.path, map_location=self.device_cpu)
        self.__print('1b', 0)
        time.sleep(2)
        self.__print('2a', 0)
        m.load_state_dict(l)
        # set to device
        self.__print('2b', 0)
        time.sleep(2)
        try:
            self.__print('3a', 0)
            m.to(self.device) # ----> This line pops up cmd
            self.__print('3b', 0)
        except RuntimeError as e:
            self.__print(str(e), 0)

When visually debugging those cmd pops up, its always in step 1 (m.load_state_dict(torch.load(self.path, map_location=self.device)))
I have tried something like disabling console output which didnt work.
import contextlib
with contextlib.redirect_stdout(None):
    ...

The if __name__=='__main__': makes no difference and also this is all part of heavy multiprocessing in some lower subprocess
Update
I traced problem to switching device - if I use torch.load(self.path, map_location=self.device_cpu) and later .to(self.device_gpu) it pops cmd on line with .to(...) but if I use torch.load(self.path, map_location=self.device_gpu) it pops on that line. Another thing to note is, that it does not matter on which device model is saved.
I am open to any workaround.

Output: Updating pytorch version by install command from their website solved the issue


Instruction: Delete duplicated rows in torch.tensor

Input:
I have a torch.tensor of shape (n,m) and I want to remove the duplicated rows (or at least find them). For example:
t1 = torch.tensor([[1, 2, 3], [4, 5, 6], [1, 2, 3], [4, 5, 6]])
t2 = remove_duplicates(t1)

t2 should be now equal to tensor([[1, 2, 3], [4, 5, 6]]), that is rows 1 and 3 are removed. Do you know a way to perform this operation?
I was thinking to do something with torch.unique but I cannot figure out what to do.

Output:
You can simply exploit the parameter dim of torch.unique.
t1 = torch.tensor([[1, 2, 3], [4, 5, 6], [1, 2, 3], [4, 5, 6], [7, 8, 9]])
torch.unique(t1, dim=0)

In this way you obtain the result you want:
tensor([[1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]])

Here you can read the meaning of that parameter.


Instruction: Manually assign weights using PyTorch

Input:
I am using Python 3.8 and PyTorch 1.7 to manually assign and change the weights and biases for a neural network. As an example, I have defined a LeNet-300-100 fully-connected neural network to train on MNIST dataset. The code for class definition is:
class LeNet300(nn.Module):
    def __init__(self):
        super(LeNet300, self).__init__()

        # Define layers-
        self.fc1 = nn.Linear(in_features = input_size, out_features = 300)
        self.fc2 = nn.Linear(in_features = 300, out_features = 100)
        self.output = nn.Linear(in_features = 100, out_features = 10)

        self.weights_initialization()

    def forward(self, x):
        out = F.relu(self.fc1(x))
        out = F.relu(self.fc2(out))
        return self.output(out)

    def weights_initialization(self):
        '''
        When we define all the modules such as the layers in '__init__()'
        method above, these are all stored in 'self.modules()'.
        We go through each module one by one. This is the entire network,
        basically.
        '''
        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.xavier_normal_(m.weight)
                nn.init.constant_(m.bias, 0)

To experiment with trying to change the weights for this model-
# Instantiate model-
mask_model = LeNet300()

To assign all of the weights in each of the layers to one (1), I use the code-
with torch.no_grad():
    for layer in mask_model.state_dict():
        mask_model.state_dict()[layer] = nn.parameter.Parameter(torch.ones_like(mask_model.state_dict()[layer]))

# Sanity check-
mask_model.state_dict()['fc1.weight']

This output shows that the weights are not equal to 1.
I also tried the code-
for param in mask_model.parameters():
    # print(param.shape)
    param = nn.parameter.Parameter(torch.ones_like(param))

But this does not work as well.
Help?

Output:
for param in mask_model.parameters():
    param.data = nn.parameter.Parameter(torch.ones_like(param))


Instruction: Linear Regression with Pytorch : constant loss

Input:
I'm working on a linear regression problem with Pytorch (y=A*x, where the dimensions of A are 2x2). I wrote the following code. I don't know why the loss doesn't change... Can someone help me ?
Thanks,
Thomas
import torch
import numpy as np
from scipy.integrate import odeint
from matplotlib import pyplot as plt
from torch.autograd import Variable

def EDP(X,t):
    X_0=-2*X[0]
    X_1=-2*X[1]
    grad=np.array([X_0,X_1])
    return grad

T=np.arange(0,10,0.1)
X_train=odeint(EDP,[10,20],T)

Y_train=np.zeros_like(X_train)
for i in range(Y_train.shape[0]):
    Y_train[i,:]=np.dot(np.array([[2,0],[0,2]]),X_train[i,:])
print(X_train,Y_train)

X_train=torch.Tensor(X_train)
torch.transpose(X_train,0,1)
Y_train=torch.Tensor(Y_train)
print(X_train.shape)

import torch.nn as nn

class LinearRegression(torch.nn.Module):

    def __init__(self):
        super(LinearRegression, self).__init__()
        self.linear = torch.nn.Linear(2,2,bias = False) # bias is default True

    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(our_model.parameters(), lr = 0.0001)

our_model = LinearRegression()

x_train = X_train
y_train = Y_train
#x_train.requires_grad=True
print(x_train.shape)
print(y_train.shape)
ntrain=10

for t in range(ntrain):

    y_pred=our_model(x_train)
    loss=criterion(y_train,y_pred)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    print(t,loss)

print(our_model.linear.weight)

Output:
In my laptop it worked ...
since you are running it on just 10 epochs ...and using lr = 0.0001 ,you wont see it in just 10 epochs.
i did this optimizer = torch.optim.SGD(our_model.parameters(), lr = 0.01) (increased lr )which actually decreased the loss in just 10 epochs


Instruction: PyTorch's grid_sample conversion to CoreML (via coremltools)

Input:
torch.nn.functional.grid_sample (source here, click on docs for documentation) is currently unsupported operation by CoreML (and their conversion utilities library: coremltools).
What I'm looking for is a way to export layer shown below from PyTorch's torchscript (docs here) to CoreML (either using custom op created via Swift or via efficient PyTorch rewrite of grid_sample).
For details and tips to get you started see Tips section
Minimal verifiable example
import coremltools as ct
import torch

class GridSample(torch.nn.Module):
    def forward(self, inputs, grid):
        # Rest could be the default behaviour, e.g. bilinear
        return torch.nn.functional.grid_sample(inputs, grid, align_corners=True)

# Image could also have more in_channels, different dimension etc.,
# for example (2, 32, 64, 64)
image = torch.randn(2, 3, 32, 32)  # (batch, in_channels, width, height)
grid = torch.randint(low=-1, high=2, size=(2, 64, 64, 2)).float()

layer = GridSample()
# You could use `torch.jit.script` if preferable
scripted = torch.jit.trace(layer, (image, grid))

# Sanity check
print(scripted(image, grid).shape)

# Error during conversion
coreml_layer = ct.converters.convert(
    scripted,
    source="pytorch",
    inputs=[
        ct.TensorType(name="image", shape=image.shape),
        ct.TensorType(name="grid", shape=grid.shape),
    ],
)

which raises the following error:
Traceback (most recent call last):
  File "/home/REDACTED/Downloads/sample.py", line 23, in <module>
    coreml_layer = ct.converters.convert(
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/_converters_entry.py", line 175, in convert
    mlmodel = mil_convert(
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 128, in mil_convert
    proto = mil_convert_to_proto(, convert_from, convert_to,
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 171, in mil_convert_to_proto
    prog = frontend_converter(, **kwargs)
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 85, in __call__
    return load(*args, **kwargs)
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 81, in load
    raise e
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 73, in load
    prog = converter.convert()
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 227, in convert
    convert_nodes(self.context, self.graph)
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 54, in convert_nodes
    raise RuntimeError(
RuntimeError: PyTorch convert function for op 'grid_sampler' not implemented.

Dependencies
Python (conda):

coremltools==4.1
torch==1.8.0

You could also use nightly/master builds (at least for the day of writing: 2021-03-20)
Tips
Those were split into two possible solutions I currently see:
PyTorch only
Rewrite torch.nn.functional.grid_sample from scratch.

This would require sticking only to PyTorch operations on tensors as loops (e.g. triple nested) would hang the converter and be too inefficient
You cannot use __getitem__ on list or related types - seems to work with torch.Tensor but had problems with that so you should have it in mind if you get RuntimeError: PyTorch convert function for op '__getitem__' not implemented

Pros:

No need for two languages & sticking to single technology

Cons:

Limited with loops and would require sticking to vectorized operations (most/all of the time)

Swift & CoreML
Register custom layer which is responsible for running grid_sample. CPU only implementation would be fine (although using Apple's Metal for GPU speedups would be great).
As I'm not into Swift, I've gathered a few resources which might help you:

https://coremltools.readme.io/docs/custom-operators - starting point, Python only, quite easy, only registering layer for conversion
https://developer.apple.com/documentation/coreml/mlcustomlayer - API of the layer which one would have to code in Swift
https://developer.apple.com/documentation/coreml/core_ml_api/creating_a_custom_layer - more about aforementioned (but not much)
https://machinethink.net/blog/coreml-custom-layers/ - blog post with example and dispatching layer to devices (GPU, CPU). Needs Swift (CPU version), Metal (GPU implementation). Eventual Metal implementation might be based off PyTorch's CUDA impl, CPU and Swift might also be related.
3 years old, so be aware of that, swish activation layer, seems to be a good starting point (and other posts from the same author cast some light on the CoreML itself).
https://github.com/hollance/CoreML-Custom-Layers - repo for the above

Pros:

Possibility to use loops and finer control over the algorithm
Might be easier as we're not limited to operations which CoreML can currently read

Cons:

Two languages
Sparse documentation

Output:
Apparently some good soul saw our struggles and provided custom op using MIL (intermediate representation language of CoreML).
Blog post where I found the solution and gist with grid sample
I am not sure why OP did not post it here, but please do respond with a comment if you want to take some SO points for your solution!
Full operation conversion code below:
from coremltools.converters.mil import register_torch_op, register_op
from coremltools.converters.mil.mil.ops.defs._op_reqs import *

# Custom operator for `torch.nn.functional.grid_sample`
@register_op(doc_str="Custom Grid Sampler", is_custom_op=True)
class custom_grid_sample(Operation):
    input_spec = InputSpec(
        x = TensorInputType(),
        grid = TensorInputType(),
        mode = StringInputType(const=True, optional=True),
        padding_mode = StringInputType(const=True, optional=True),
        align_corners = BoolInputType(const=True, optional=True)
    )

    bindings = {
        "class_name": "CustomGridSampler",
        "input_order": ["x", "grid"],
        "parameters": ["mode", "padding_mode", "align_corners"],
        "description": "Custom Grid Sampler"
    }

    def __init__(self, **kwargs):
        super(custom_grid_sample, self).__init__(**kwargs)

    def type_inference(self):
        x_type = self.x.dtype
        x_shape = self.x.shape

        grid_type = self.grid.dtype
        grid_shape = self.grid.shape

        assert len(x_shape) == len(grid_shape) == 4
        assert grid_shape[-1] == 2

        shape = list(x_shape)
        shape[-2] = grid_shape[1]
        shape[-1] = grid_shape[2]
        return types.tensor(x_type, tuple(shape))

@register_torch_op
def grid_sampler(context, node):
    inputs = _get_inputs(context, node)
    x = inputs[0]
    grid = inputs[1]
    mode = node.attr.get("mode", "bilinear")
    padding_mode = node.attr.get("padding_mode", "zeros")
    align_corners = node.attr.get("align_corners", False)
    x = mb.custom_grid_sample(
        x=x,
        grid=grid,
        mode=mode,
        padding_mode=padding_mode,
        align_corners=align_corners,
        name=node.name
    )
    context.add(x)


Instruction: pytorch loss accumulated when using mini-batch

Input:
I am new to pytorch. May I ask what is the difference between adding 'loss.item()' or not? The following 2 parts of code:
for epoch in range(epochs):
    trainingloss =0
    for i in range(0,X.size()[1], batch_size):
        indices = permutation[i:i+batch_size]
        F = model.forward(X[n])
        optimizer.zero_grad()
        criterion = loss(X,n)
        criterion.backward()
        optimizer.step()
        trainingloss += criterion.item()

and this
for epoch in range(epochs):
    for i in range(0,X.size()[1], batch_size):
        indices = permutation[i:i+batch_size]
        F = model.forward(X[n])
        optimizer.zero_grad()
        criterion = loss(X,n)
        criterion.backward()
        optimizer.step()

If anyone has any idea please help. Thank you very much.

Output:
Calling loss.item() allows you to take a loss variable that is detached from the computation graph that PyTorch creates (this is what .item() does for PyTorch variables).
If you add the line trainingloss += criterion.item() at the end of each "batch loop", this will keep track of the batch loss throughout the iteration by incrementally adding the loss for each minibatch in your training set.  This is necessary since you are using minibatches - the loss for each minibatch will not be equal to the loss over all the batches.
Note: If you use PyTorch variables outside the optimization loop, e.g. in a different scope, which could happen if you call something like return loss, it is crucial that you call .item() on any PyTorch variables that are part of the computation graph (as a general rule of thumb, any outputs/loss/models that interact with PyTorch methods will likely be part of your computation graph). If not, this can cause the computation graph to not be de-allocated/deleted from Python memory, and can lead to CPU/GPU memory leaks.  What you have above looks correct though!
Also, in the future, PyTorch's DataLoader class can help you with minibatches with less boilerplate code - it can loop over your dataset such that each item you loop over is a training batch - i.e. you don't require two for loops in your optimization.
I hope you enjoy learning/using PyTorch!


Instruction: How gradients are accumulated in real

Input:
‘Gradient will not be updated but be accumulated, and updated every N rounds.’ I have a question that how the gradients are accumulated in the below code snippet: in every round of the below loop I can see a new gradient is computed by loss.backward() and should be stored internally, but would this internally stored gradient be refreshed in the next round? How the gradient is summed up, and later be applied every N rounds?
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()

Output:
The first time you call backward, the .grad attribute of the parameters of your model will be updated from None, to the gradients. If you do not reset the gradients to zero, future calls to .backward() will accumulate (i.e. add) gradients into the attribute (see the docs).
When you call model.zero_grad() you are doing the reset.


Instruction: Predictions doesn't equal number of images

Input:
My validation data are 150 images, but when i try to use my model to predict them my predictions are of length 22 I don't understand why?
total_v=0
correct_v=0

with torch.no_grad():
    model.eval()
    for data_v, target_v in (validloader):
        if SK:
            target_v = torch.tensor(np.where(target_v.numpy() == 2, 1, 0).astype(np.longlong))
        else:
            target_v = torch.tensor(np.where(target_v.numpy() == 0, 1, 0).astype(np.longlong))

        data_v, target_v = data_v.to(device), target_v.to(device)

        outputs_v = model(data_v)
        loss_v = criterion(outputs_v, target_v)
        batch_loss += loss_v.item()
        _,pred_v = torch.max(outputs_v, dim=1)
        correct_v += torch.sum(pred_v==target_v).item()
        total_v += target_v.size(0)
    val_acc.append(100 * correct_v/total_v)
    val_loss.append(batch_loss/len(validloader))
    network_learned = batch_loss < valid_loss_min
    print(f'validation loss: {np.mean(val_loss):.4f}, validation acc: {(100 *
          correct_v/total_v):.4f}\n')

this is my model
model = models.resnet50(pretrained = True)
num_ftrs = model.fc.in_features

model.fc = nn.Linear(num_ftrs, 2)

model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adagrad(model.parameters())

Output:
If you want to have the whole predictions you should store predictions of each individual batch and concatenate them at the end of iterations
...
all_preds = []

for data_v, target_v in validloader:
    ....
     _,pred_v = torch.max(outputs_v, dim=1)
    all_preds.append(pred_v)
    ....

all_preds = torch.cat(all_preds).cpu().numpy()
print(len(all_preds))


Instruction: len() vs .size(0) when looping through DataLoader samples

Input:
I came across this on github (snippet from here):
(...)
for epoch in range(round):
        for i, data in enumerate(dataloader, 0):
            ############################
            # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
            ###########################
            # train with real
            netD.zero_grad()
            real_cpu = data[0].to(device)
            batch_size = real_cpu.size(0)
            label = torch.full((batch_size,), real_label, device=device)
            (...)

Would replacing batch_size = real_cpu.size(0) with batch_size = len(data[0]) give the same effect? (or maybe at least with batch_size = len(real_cpu)?) Reason why I'm asking is that iirc the official PyTorch tutorial incorporated len(X) when displaying training progress during the loop for (X, y) in dataloader: etc. so I was wondering if the two methods are equivalent for displaying the number of 'samples' in the 'current' batch.

Output:
If working with data where batch size is the first dimension then you can interchange real_cpu.size(0) with len(real_cpu) or with len(data[0]).
However when working with some models like LSTMs you can have batch size at second dimension, and in such case you couldn't go with len, but rather real_cpu.size(1) for example


Instruction: Predicting a Chess Board Position using Pytorch

Input:
I want to predict a the current chess board using pytorch/keras. (Let's not worry about the input for now.)
How would I got about that?
A chess board has 8x8 positions (64) on each position could be a black or white piece (12) or no piece at all (1). I am planning on using this representation for the chess board (other suggestions are welcome!):
https://en.wikipedia.org/wiki/Board_representation_(computer_chess)#Square_list
For example:
 2 3 4 5 6 4 3 2
 1 1 1 1 1 1 1 1
 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0
-1-1-1-1-1-1-1-1
-2-3-4-5-6-4-3-2`

As far as I know it is not possible to predict something like this. Because the number of classes my final layer would have to predict are 448 (64x7) and I don't feel like a NN could do that. Additionally there is the problem that softmax wouldn't work (imo). Also the Loss function might become a problem as well.
Does someone have an idea on how to do that? Or could point me in the right direction, because multi-class classification isn't really the right term for this task. I was thinking about creating 6 networks that create a classification for each piece. So a 8x8 array that looks like this (for rooks):
 10000001
 00000000
 00000000
 00000000
 00000000
-1000000-1

But the problem is still quite similar.
I think creating 64 NNs that take care of one position each would simplify the problem a bit. But that would be a pain to train.
Looking forward to hearing your suggestions!

Output:
For anyone wondering how to do this. I think I figured it out:
You build a Softmax over the third dimension of a 8x8x13 array and get a 8x8 matrix with all the chess figures.
Thanks to @Prune. I will adapt my questions in the future.


Instruction: Image classification model works with 32x32 images but not 64x64

Input:
I'm trying to build a deep learning food classification algorithm using the food 101 dataset. I was able to successfully implement it by using the following model which works when all my images are sized with dimensions 32x32. However, I realised some of the images were almost incomprehensible so I increased the size to 64x64 for all images. However, when I run my code with these larger image sizes it no longer works.
I believe the error is to do with how I've defined the model. I'm new to the area of deep learning and would appreciate any help. If you need any further info pls comment below without taking down the post.
Model definition (uses convolutional layers and a residual block):
class SimpleResidualBlock(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, stride=1, padding=1)
        self.relu1 = nn.ReLU()
        self.conv2 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, stride=1, padding=1)
        self.relu2 = nn.ReLU()

    def forward(self, x):
        out = self.conv1(x)
        out = self.relu1(out)
        out = self.conv2(out)
        return self.relu2(out) + x # ReLU can be applied before or after adding the input

def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))

class ImageClassificationBase(nn.Module):
    def training_step(self, batch):
        images, labels = batch
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        return loss

    def validation_step(self, batch):
        images, labels = batch
        out = self(images)                    # Generate predictions
        loss = F.cross_entropy(out, labels)   # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss.detach(), 'val_acc': acc}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], last_lr: {:.5f}, train_loss: {:.4f}, val_loss: {:.4f}, val_acc: {:.4f}".format(
            epoch, result['lrs'][-1], result['train_loss'], result['val_loss'], result['val_acc']))

def conv_block(in_channels, out_channels, pool=False):
    layers = [nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
              nn.BatchNorm2d(out_channels),
              nn.ReLU(inplace=True)]
    if pool: layers.append(nn.MaxPool2d(2))
    return nn.Sequential(*layers)

class ResNet9(ImageClassificationBase):
    def __init__(self, in_channels, num_classes):
        super().__init__()

        self.conv1 = conv_block(in_channels, 64)
        self.conv2 = conv_block(64, 128, pool=True)
        self.res1 = nn.Sequential(conv_block(128, 128), conv_block(128, 128))

        self.conv3 = conv_block(128, 256, pool=True)
        self.conv4 = conv_block(256, 512, pool=True)
        self.res2 = nn.Sequential(conv_block(512, 512), conv_block(512, 512))

        self.classifier = nn.Sequential(nn.MaxPool2d(4),
                                        nn.Flatten(),
                                        nn.Dropout(0.2),
                                        nn.Linear(512, num_classes))

    def forward(self, xb):
        out = self.conv1(xb)
        out = self.conv2(out)
        out = self.res1(out) + out
        out = self.conv3(out)
        out = self.conv4(out)
        out = self.res2(out) + out
        out = self.classifier(out)
        return out

Error I get when executing it:
RuntimeError: mat1 dim 1 must match mat2 dim 0

Output:
Welcome to stackoverflow. In general, when you give your code and the error you are receiving, it is better to at least provide a test case scenario where others can reproduce the error you are getting. In this case, I was able to identify the problem and create a test case example.
"RuntimeError: mat1 dim 1 must match mat2 dim 0" this error sounded like a matrix multiplication error to me, where you multiply two matrices and dimensions don't match for multiplication. When I look at your code, only place I see that uses a matrix multiplication is that part:
self.classifier = nn.Sequential(nn.MaxPool2d(4),
                                nn.Flatten(),
                                nn.Dropout(0.2),
                                nn.Linear(512, num_classes))

Linear layer is just a basic matrix multiplication: out = input * weight + bias. So it looks like input dimension of linear layer and weight matrix don't match when you change the input image size:
 model_resnet = ResNet9(3, 10)
 img = torch.rand(10, 3, 64, 64)
 out = model_resnet(img)

Reason this happens is that you use MaxPool2d(4) which applies a 4x4 max pooling filter over the input. If input dimensions to max pooling is 4x4, this filter will produce 1x1 result, if it is 8x8 it will produce 2x2 output, so when you increase your input dimensions to 64x64 from 32x32, output of max pooling will be doubled in each axis making nn.Linear(512) not suitable for your new dimension.
Solution is simple, use Adaptive pooling operations. For example:
self.classifier = nn.Sequential(nn.AdaptiveAvgPool2d((1,1)),
                                nn.Flatten(),
                                nn.Dropout(0.2),
                                nn.Linear(512, num_classes))

AdaptiveAvgPool2d will apply a average filter over the received input, and produce 1x1 result all the time no matter the input dimension. Essentially if the input is 8x8 it will apply 8x8 averaging filter, if the input is 4x4 it will apply 4x4 averaging filter. So with this simple change, you can use 32x32 and 64x64 and even higher dimensional images.


Instruction: How to concatenate tensor to another list of tensor in pytorch?

Input:
I have a tensor of shape "torch.Size([2, 2, 3])" and another tensor of shape "torch.Size([2, 1, 3])". I want a concatenated tensor of shape "torch.Size([2, 2, 6])".
 For example :
a=torch.tensor([[[2,3,5],[12,13,15]],[[20,30,50],[120,130,150]]])
b=torch.tensor([[[99,99,99]],[[999,999,999]]])

I want the output as : [[[99,99,99,2,3,5],[99,99,99,12,13,15]],[[999,999,999,20,30,50],[999,999,999,120,130,150]]]
I have written a O(n2) solution using two for loops but,
This is taking a lot of time with millions of calculation, Does anyone help me in doing this efficiently ?? May be some matrix calculation trick for tensors ??

Output:
To exactly match the example you have provided:
c = torch.cat([b.repeat([1,a.shape[1]//b.shape[1],1]),a],2)
The reasoning behind this is that the concatenate operation in pytorch (and numpy and other libraries) will complain if the dimensions of the two tensors in the non-specified axes (in this case 0 and 1) do not match. Therefore, you have to repeat the tensor along the non-matching axis (the first axis, therefore the second element of the repeat list) in order to make the dimensions align. Note that the solution here will only work if the middle dimension of a is evenly divisible by the middle dimension of b.
In newer versions of pytorch, this can also be done using the torch.tile() function.


Instruction: Running deep-daze on nvidia jetson - "Could not find a version that satisfies the requirement torchvision>=0.8.2"

Input:
I have an nvidia Jetson Nano (the 4gb version).  I am attempting to run this project on it: https://github.com/lucidrains/deep-daze
I am attempting to run the command pip install deep-daze.  However, I do not have pip so I am running pip3 install deep-daze.  When I run that I get
chris@chris-desktop:~$ pip3 install deep-daze
Collecting deep-daze
  Using cached https://files.pythonhosted.org/packages/f1/ed/b3f3d9d92f5a48932b3807f683642b28da75722ae93da2f9bdc6af5f1768/deep_daze-0.7.2-py3-none-any.whl
Collecting tqdm (from deep-daze)
  Downloading https://files.pythonhosted.org/packages/f8/3e/2730d0effc282960dbff3cf91599ad0d8f3faedc8e75720fdf224b31ab24/tqdm-4.59.0-py2.py3-none-any.whl (74kB)
    100% |████████████████████████████████| 81kB 2.4MB/s
Collecting torchvision>=0.8.2 (from deep-daze)
  Could not find a version that satisfies the requirement torchvision>=0.8.2 (from deep-daze) (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3)
No matching distribution found for torchvision>=0.8.2 (from deep-daze)

I am pretty unfamiliar with several of the moving parts here and not sure how to fix this.  I thought these version numbers may be useful in answering this question:
chris@chris-desktop:~$ python3 --version
Python 3.6.9
chris@chris-desktop:~$ pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
chris@chris-desktop:~$ python2 --version
Python 2.7.17
chris@chris-desktop:~$ pip2 --version
bash: pip2: command not found
chris@chris-desktop:~$ pip --version
bash: pip: command not found
chris@chris-desktop:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Output:
Thanks to the comment from Elbek I got this working!  I was able to follow the guide here: https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-8-0-now-available/72048
Unfortunately after I got everything installed I ran into an issue with not having enough memory, but, it all got installed.