Instruction: How to measure performance of a pretrained HuggingFace language model?

Input:
I am pretraining a GPT2LMHeadModel using Trainer as follows:
training_args = TrainingArguments(
    output_dir=str(project_root / 'models/bn-gpt2/'),
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=4,
    fp16=True,
    optim="adafactor",
    eval_steps=400,
    save_steps=800,
    warmup_steps=500,
    evaluation_strategy="steps",
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=tokenized_dataset['train'],
    eval_dataset=tokenized_dataset['test'],
)

trainer.train()

I want to measure the performance of my pre-trained model using perplexity or accuracy metrics during and after training. I have found some ways to measure these for individual sentences, but I cannot find a way to do this for the complete model. My goal is to create a next word prediction model for my native language using GPT2 training from scratch.

Output:
If I understand it correctly then this tutorial shows how to calculate perplexity for the entire test set. If I see it correctly they use the entire test corpus as one string connected by linebreaks, which might have to do with the fact that perplexity uses a sliding window which uses the text that came previous in the corpus. I personally did not calculate perplexity for a model yet and am not an expert at this. In any case you could average the sentence score into a corpus score, although there might be issues with the logic of how that metric works as well as the weighting since sentences can have a different number of words, see this explaination.
Also I'm not sure if you are already aware of this but there is also a pretrained GPT-2 model available for Bengali on huggingface.


Instruction: Unable to train PyTorch model in GPU. Keep getting errors that tensors are not on same device

Input:
I have been stuck at trying to train my PyTorch model in GPU. The model perfectly works in CPU though. I have been using Google Colab's GPU resources for using cuda.
I know that in order to run a model in GPU, the 'model', 'input features' and 'target' needs to be in 'cuda' device.
But, no matter what I do in my code, I either keep getting the error:
RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu

OR
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Here is my notebook:
https://colab.research.google.com/drive/1rviS_4hmdzPQUncZyi8FsRH7y3jL0isQ
It would be really helpful if someone could let me exactly which variables to be moved using .to('cuda')
Additionally, explanations/suggestions for ensuring that this does not recur in the future would be highly appreciated. Thank you !

Output:
Your self.hidden is a tuple of torch.tensors. PyTorch doesn't automatically move these kind of tensor to GPU when .to(device) is invoked on your model.
You can either:

Implement your own to(self, type, device) method for your BiLSTM_CRF class. (Not recommended).
Make self.hidden a registered buffer. This way all methods of nn.Module such as .to(), .float(), etc. will also be applied to self.hidden.


Instruction: Why am I getting different results when I use models with the same weights in different formats - \(.pt) \.onnx \(.bin, .xml)?

Input:
I have a model trained on YOLOv5s and is working fine.
This is an input image:

I can get an expected result using pytorch after doing an inference:

This is an output image:

The thing is, I need it in Openvino and regardless if I do the inference using the model in .onnx or .bin and .xml (for openvino) I won't get the expected inference result.
What I get is a vector with this shape (1, 25200, 6).
I know that:

25200 is equal to 1x3x80x80 + 1x3x40x40 + 1x3x20x20;
6 = 1 class + 4 (x,y,w,h) + 1 (score);
batch_size = 1

To export it, I used:
!python export.py --data models/custom_yolov5s.yaml --weights /content/bucket_11_03_2022.pt --batch-size 1 --device cpu --include openvino --imgsz 640

and to reproduce the issue I did in two ways:

.onnx:

import cv2
image = cv2.imread('data/cropped.png')

# Resize image to meet network expected input sizes
resized_image = cv2.resize(image, (640, 640))

# Reshape to network input shape
input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0)

import onnxruntime as onnxrt

onnx_session= onnxrt.InferenceSession("models/bucket_11_03_2022.onnx")
onnx_inputs= {onnx_session.get_inputs()[0].name:input_image.astype(np.float32)}
onnx_output = onnx_session.run(None, onnx_inputs)
img_label = onnx_output[0]
print(onnx_output[0].shape)

Openvino:

import cv2
import matplotlib.pyplot as plt
import numpy as np
from openvino.inference_engine import IECore

ie = IECore()

net = ie.read_network(
    model="bucket_11_03_2022.xml",
    weights="bucket_11_03_2022.bin",
)
exec_net = ie.load_network(net, "CPU")

output_layer_ir = next(iter(exec_net.outputs))
input_layer_ir = next(iter(exec_net.input_info))

# Text detection models expects image in BGR format
image = cv2.imread("data/cropped.png")

# N,C,H,W = batch size, number of channels, height, width
N, C, H, W = net.input_info[input_layer_ir].tensor_desc.dims

# Resize image to meet network expected input sizes
resized_image = cv2.resize(image, (W, H))

# Reshape to network input shape
input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0)

plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB));

result = exec_net.infer(inputs={input_layer_ir: input_image})

result['output'].shape

Could you guys help me to get the correct inference (bounding box with score) using .onnx or the IE format (openvino - .bin, .xml)?
The model files are here.

Output:
Based on my replication, this issue occurred due to incorrect conversion from PyTorch to ONNX. I’ve found that the converted ONNX from the PyTorch model was able to detect the object (bucket) but did not reflect the correct label as it took one of the class names from coco128.yaml.
You may need to retrain your model by following the Train Custom Data. But I cannot guarantee this method will be successful as it is not validated by OpenVINO.
I suggest you post this issue in ultralytics GitHub forum. For your information, ultralytics is not a part of OpenVINO Toolkit.


Instruction: How to add data augmentation with albumentation to image classification framework?

Input:
I am using pytorch for image classification using this code from github.
I need to add data augmentation before training my model,
I chose albumentation to do this.
here is my code when I add albumentation:
data_transform = {
    "train": A.Compose([
                        A.RandomResizedCrop(224,224),
                        A.HorizontalFlip(p=0.5),
                        A.RandomGamma(gamma_limit=(80, 120), eps=None, always_apply=False, p=0.5),
                        A.RandomBrightnessContrast (p=0.5),
                        A.CLAHE(clip_limit=4.0, tile_grid_size=(8, 8), always_apply=False, p=0.5),
                        A.ShiftScaleRotate(shift_limit=0.05, scale_limit=0.05, rotate_limit=15, p=0.5),
                        A.RGBShift(r_shift_limit=15, g_shift_limit=15, b_shift_limit=15, p=0.5),
                        A.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
                        ToTensorV2(),]),
    "val": A.Compose([
                      A.Resize(256,256),
                      A.CenterCrop(224,224),
                      A.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
                      ToTensorV2()])}

I got this error:

KeyError: Caught KeyError in DataLoader worker process 0.
KeyError: 'You have to pass data to augmentations as named arguments, for example: aug(image=image)'

Output:
This Albumentations function takes a positional argument 'image' and returns a dictionnary. This is a sample to use it :
transforms = A.Compose([
                A.augmentations.geometric.rotate.Rotate(limit=15,p=0.5),
                A.Perspective(scale=[0,0.1],keep_size=False,fit_output=False,p=1),
                A.Resize(224, 224),
                A.HorizontalFlip(p=0.5),
                A.GaussNoise(var_limit=(10.0, 50.0), mean=0),
                A.RandomToneCurve(scale=0.5,p=1),
                A.Normalize(mean=[0.5, 0.5, 0.5],std=[0.225, 0.225, 0.225]),
                ToTensorV2()
            ])

img = cv2.imread("dog.png")
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
transformed_img = transforms(image=img)["image"]


Instruction: Why I get "RuntimeError: CUDA error: the launch timed out and was terminated" when using Google Cloud compute engine

Input:
I have a Google cloud compute engine with 4 Nvidia K80 GPU and Ubuntu 20.04 (python 3.8). When I try to train the yolo5 model, I get the following error:
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[W CUDAGuardImpl.h:113] Warning: CUDA warning: the launch timed out and was terminated (function destroyEvent)
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:1230 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f62be2c17d2 in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x239de (0x7f62f6ea69de in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x22d (0x7f62f6ea857d in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x300568 (0x7f63736d9568 in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #4: c10::TensorImpl::release_resources() + 0x175 (0x7f62be2aa005 in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #5: std::vector<c10d::Reducer::Bucket, std::allocator<c10d::Reducer::Bucket> >::~vector() + 0x2e9 (0x7f62fa8ca5e9 in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::Reducer::~Reducer() + 0x205 (0x7f62fa8bcd25 in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #7: std::_Sp_counted_ptr<c10d::Reducer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x12 (0x7f6373bb7212 in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #8: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x46 (0x7f63735c7506 in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x7e182f (0x7f6373bba82f in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #10: <unknown function> + 0x1f5b20 (0x7f63735ceb20 in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x1f6cce (0x7f63735cfcce in /home/cheyuxuanll/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #12: /usr/bin/python3() [0x5d1ec4]
frame #13: /usr/bin/python3() [0x5a958d]
frame #14: /usr/bin/python3() [0x5ed1a0]
frame #15: /usr/bin/python3() [0x544188]
frame #16: /usr/bin/python3() [0x5441da]
frame #17: /usr/bin/python3() [0x5441da]
frame #18: PyDict_SetItemString + 0x538 (0x5ce7c8 in /usr/bin/python3)
frame #19: PyImport_Cleanup + 0x79 (0x685179 in /usr/bin/python3)
frame #20: Py_FinalizeEx + 0x7f (0x68040f in /usr/bin/python3)
frame #21: Py_RunMain + 0x32d (0x6b7a1d in /usr/bin/python3)
frame #22: Py_BytesMain + 0x2d (0x6b7c8d in /usr/bin/python3)
frame #23: __libc_start_main + 0xf3 (0x7f6378be40b3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #24: _start + 0x2e (0x5fb12e in /usr/bin/python3)

I am training this model with this command:
python3 -m torch.distributed.run  --nproc_per_node 4 train.py --batch 16 --data coco128.yaml --weights yolov5s.pt --device 0,1,2,3

Am I missing something here?
Thanks

Output:
We are also running CUDA in the Google Cloud and our server restarted roughly when you posted your question. While we couldn't detect any changes, our service couldn't start due to "RuntimeError: No CUDA GPUs are available".
So there are some similarities, but also some differences.
Anyway, we opted for the good ol' uninstall and reinstall and that fixed it:
Uninstall:
sudo apt-get --purge remove "*cublas*" "cuda*" "nsight*"
sudo apt-get --purge remove "*nvidia*"

Plus deleting anything in /usr/local/*cuda*
Install:
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/ /"
sudo add-apt-repository contrib
sudo apt-get update
sudo apt-get -y install cuda-11-3

We also reinstalled CUDNN, but that may or may not be part of your stack.


Instruction: can't import torchtext.legacy.data

Input:
as i know, from torchtext 0.9.0, torchtext.data and torchtext.dataset are moved to torchtext.legacy
but my 0.12.0 torchtext can't import torchtext.legacy
while it can import torchtext.data
I tried if it moved to torchtext.data again but I can't find any document

torch.version == 1.11.0

Output:
I also faced the same problem wtih the same versions. The only thing I was able to do about it is to install previous version of torchtext:
pip install torchtext==0.6.0

Only then was I wable to import the packs.


Instruction: Pytorch RuntimeError: CUDA out of memory with a huge amount of free memory

Input:
While training the model, I encountered the following problem:
RuntimeError: CUDA out of memory. Tried to allocate 304.00 MiB (GPU 0; 8.00 GiB total capacity; 142.76 MiB already allocated; 6.32 GiB free; 158.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
As we can see, the error occurs when trying to allocate 304 MiB of memory, while 6.32 GiB is free! What is the problem? As I can see, the suggested option is to set max_split_size_mb to avoid fragmentation. Will it help and how to do it correctly?
This is my version of PyTorch:
torch==1.10.2+cu113
torchvision==0.11.3+cu113
torchaudio===0.10.2+cu113

Output:
I tried hours til i found out:
to reduce the  batch size
and the resize my input image  image size


Instruction: How to train MLM model XLM Roberta large on google machine specs fastly with less memory

Input:
I am fine tuning masked language model from XLM Roberta large on google machine specs.
I made couple of experiments and was strange to see few results.
"a2-highgpu-4g" ,accelerator_count=4, accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 4 Running ( 4 data*4 GPU=16 data points)
"a2-highgpu-4g" ,accelerator_count=4 , accelerator_type="NVIDIA_TESLA_A100"on 4,12,672 data batch size 8 failed
 "a2-highgpu-4g" ,accelerator_count=4, accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 16 failed
"a2-highgpu-4g" ,accelerator_count=4.,accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 32 failed

I was not able to train model with batch size  more than 4 on # of GPU's. It stopped in mid-way.
Here is the code I am using.
training_args = tr.TrainingArguments(
#     disable_tqdm=True,
    output_dir='/home/pc/Bert_multilingual_exp_TCM/results_mlm_exp2',
    overwrite_output_dir=True,
    num_train_epochs=2,
    per_device_train_batch_size=4,
#     per_device_train_batch_size
#     per_gpu_train_batch_size
    prediction_loss_only=True
    ,save_strategy="no"
    ,run_name="MLM_Exp1"
    ,learning_rate=2e-5
    ,logging_dir='/home/pc/Bert_multilingual_exp_TCM/logs_mlm_exp1'        # directory for storing logs
    ,logging_steps=40000
    ,logging_strategy='no'
)

trainer = tr.Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_data

)

My Questions
How can I train with larger batch size on a2-highgpu-4g machine?
Which parameters can I include in TrainingArguments so that training is fast and occupies small memory?
Thanks in advance.
Versions
torch==1.11.0+cu113

torchvision==0.12.0+cu113

torchaudio==0.11.0+cu113

transformers==4.17.0

Output:
I was facing a similar dilemma some days ago when I came across this enlightening article from Hugginface.
I did the experiments myself and could see the improvements in training. As I'm working with a Tesla T4, the following configuration allows me to resume training:
training_args = TrainingArguments(
  output_dir=f'./{CN_MODEL_NAME}',
  overwrite_output_dir=True,
  num_train_epochs=2,
  per_device_train_batch_size=8,
  gradient_accumulation_steps=1,
  gradient_checkpointing=True,
  optim="adafactor",
  save_steps=1000,
  save_total_limit=1,
  warmup_steps=1000,
  weight_decay=0.01,
  learning_rate=1e-5,
  report_to=["wandb"],
  logging_steps=500,
  do_eval=False,
  fp16=True
)

Anything greater than that led me to the dreaded CUDA Out of memory.
Hope it works for you too.


Instruction: Why, using Huggingface Trainer, single GPU training is faster than 2 GPUs?

Input:
I have a VM with 2 V100s and I am training gpt2-like models (same architecture, fewer layers) using the really nice Trainer API from Huggingface. I am using the pytorch back-end.
I am observing that when I train the exact same model (6 layers, ~82M parameters) with exactly the same data and TrainingArguments, training on a single GPU training is significantly faster than on 2GPUs: ~5hrs vs ~6.5hrs.
How would one debug this kind of issue to uderstand what's causing the slowdown?
Extra notes:

the 2 gpus are both being used (watching nvidia-smi output)
I am using fp16 precision
My TrainingArguments values are:

{
    "optim": "adamw_torch",
    "evaluation_strategy": "epoch",
    "save_strategy": "epoch",
    "fp16": true,
    "gradient_checkpointing": true,
    "per_device_train_batch_size": 16,
    "per_device_eval_batch_size": 16,
    "dataloader_num_workers": 4,
    "dataloader_pin_memory": true,
    "gradient_accumulation_steps": 1,
    "num_train_epochs": 5
}

The output of nvidia-smi topo -m is:
$ nvidia-smi topo -m
        GPU0    GPU1    CPU Affinity    NUMA Affinity
GPU0     X      SYS     0-11            N/A
GPU1    SYS      X      0-11            N/A

I understand that without NVLink inter-gpu communication is not as fast as it could be, but can that be the only cause of a slowdown like the one I'm observing? And if so, is there anything I can do or will I always have slower training times on 2GPUs (thus making multi-gpu training essentially useless)?

Output: Keeping this here for reference. The cause was   "gradient_checkpointing": true,. The slowdown induced by gradient checkpointing appears to be larger on 2 GPUs than on a single GPU. I don't really know the cause of this issue, if anyone knows I would really appreaciate someone telling me.


Instruction: pip does not find the cudatoolkit that conda has installed

Input:
I'm trying to install torch_scatter with pip. However it gives me an error message:
      File "/home1/huangjiawei/miniconda3/envs/lin/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 404, in build_extensions
        self._check_cuda_version()
      File "/home1/huangjiawei/miniconda3/envs/lin/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 781, in _check_cuda_version
        raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
    RuntimeError:
    The detected CUDA version (9.0) mismatches the version that was used to compile
    PyTorch (11.3). Please make sure to use the same CUDA versions.

But i did install cudatoolkit by conda:
(lin) huangjiawei@ai-server-2:~/linzhijie_Weakly-supervised-Query-based-Video-Segmentation$ conda list|grep cuda
cudatoolkit               11.3.1               h2bc3f7f_2    defaults
pytorch                   1.10.2          py3.8_cuda11.3_cudnn8.2.0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
pytorch-mutex             1.0                        cuda    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch

So it seems that pip only detected the cuda version of the sever,but didn't detected the cuda version in my enviroment.
How to fix it?

Output:
This error is complaining that your system CUDA compiler (nvcc) version doesn't match. cudatoolkit you installed in conda is CUDA runtime. These two are different components.
To install CUDA compiler, you need to install the CUDA toolkit from NVIDIA


Instruction: Pytorch loss is nan

Input:
I'm trying to write my first neural network with pytorch.
Unfortunately, I encounter a problem when I want to get the loss.
The following error message:
RuntimeError: Function 'LogSoftmaxBackward0' returned nan values in its 0th output.

So I tried debugging and found something strange.
The input has no nans and infs as I verify with the following:
print(torch.any(torch.isnan(inputs)))

But if I always let the individual steps in the model x be output, I see that there will be inf at some point.
training
inputs, labels = data
print(torch.any(torch.isnan(inputs)))
optimizer.zero_grad()
outputs = model(inputs)
print(outputs)
loss = criterion(outputs, labels)
print(f"epoch: {epoch + 1} loss: {loss.item()}")
loss.backward()

optimizer.step()

model
class Net(Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer1 = Conv1d(in_channels=1, out_channels=5, kernel_size=5, stride=2, dtype=torch.float64)
        self.act1 = ReLU()
        self.pool1 = MaxPool1d(2)
        self.layer2 = Conv1d(in_channels=5, out_channels=1, kernel_size=2, dtype=torch.float64)
        self.fcl1 = Linear(1350, 16, dtype=torch.float64)

    def forward(self, x):
        print("raw", x)
        x = self.layer1(x)
        print("conv1d 1", x)
        x = self.act1(x)
        print("relu", x)
        x = self.layer2(x)
        print("conv1d 2", x)
        x = self.pool1(x)
        x = self.pool1(x)
        x = self.pool1(x)
        x = self.pool1(x)
        x = self.pool1(x)
        x = self.pool1(x)
        x = self.pool1(x)
        print("pools", x)
        x = self.fcl1(x)
        print("linear", x)
        return x

output
tensor(False)
raw tensor([[9.0616e+227, 2.4353e-152,  1.0294e-71,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00]], dtype=torch.float64)
conv1d 1 tensor([[   -inf,    -inf,    -inf,  ..., -0.2516, -0.2516, -0.2516],
        [    inf,     inf,     inf,  ...,  0.3377,  0.3377,  0.3377],
        [   -inf,    -inf,    -inf,  ...,  0.4285,  0.4285,  0.4285],
        [   -inf,    -inf,    -inf,  ..., -0.1230, -0.1230, -0.1230],
        [    inf,     inf,     inf,  ...,  0.3793,  0.3793,  0.3793]],
       dtype=torch.float64, grad_fn=<SqueezeBackward1>)
relu tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
        [   inf,    inf,    inf,  ..., 0.3377, 0.3377, 0.3377],
        [0.0000, 0.0000, 0.0000,  ..., 0.4285, 0.4285, 0.4285],
        [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
        [   inf,    inf,    inf,  ..., 0.3793, 0.3793, 0.3793]],
       dtype=torch.float64, grad_fn=<ReluBackward0>)
conv1d 2 tensor([[        -inf,         -inf,         -inf,  ..., -5.4167e+265,
         -5.4167e+265, -5.4167e+265]], dtype=torch.float64,
       grad_fn=<SqueezeBackward1>)
pools tensor([[        -inf, -5.4167e+265, -5.4167e+265,  ..., -5.4167e+265,
         -5.4167e+265, -5.4167e+265]], dtype=torch.float64,
       grad_fn=<SqueezeBackward1>)
linear tensor([[inf, inf, -inf, -inf, -inf, inf, inf, inf, inf, inf, inf, -inf, inf, inf, -inf, -inf]],
       dtype=torch.float64, grad_fn=<AddmmBackward0>)
tensor([[inf, inf, -inf, -inf, -inf, inf, inf, inf, inf, inf, inf, -inf, inf, inf, -inf, -inf]],
       dtype=torch.float64, grad_fn=<AddmmBackward0>)
epoch: 1 loss: nan

Thanks for helping

Output: Sorry, my reputation is not enough for me to comment directly. This may be caused by the exploding gradient due to the excessive learning rate. It is recommended that you reduce the learning rate or use weight_decay.


Instruction: An error with Omegaconf when running continuous image generation code

Input:
I found this author's PiggybackGAN code on Github (about continuous learning image generation)
The link below: https://github.com/kaushik333/Piggyback-GAN-Pytorch
The github issue has this problem,but no one has solved.
I want to run this code in my Linux environment. After configuring the environment and data set, I get the following error:
initialize network with normal
initialize network with normal
initialize network with normal
initialize network with normal
Length of loader is  10
learning rate 0.0002000 -> 0.0002000
save image!
Length of loader is  10
learning rate 0.0002000 -> 0.0002000
save image!
...
...
learning rate 0.0000040 -> 0.0000020
save image!
Length of loader is  10
learning rate 0.0000020 -> 0.0000000
save image!
Traceback (most recent call last):
  File "/opt/data/private/Pig/Piggyback-GAN-Pytorch-main/pb_cycleGAN.py", line 231, in main
    mp.spawn(train, nprocs=len(opt.gpu_ids), args=(opt,))
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/opt/data/private/Pig/Piggyback-GAN-Pytorch-main/pb_cycleGAN.py", line 88, in train
    opt.netG_A_filter_list.append([layer.unc_filt.detach().cpu()])
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/listconfig.py", line 228, in append
    self._format_and_raise(key=index, value=item, cause=e)
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/listconfig.py", line 224, in append
    parent=self,
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/omegaconf.py", line 770, in _maybe_wrap
    ref_type=ref_type,
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/omegaconf.py", line 714, in _node_wrap
    ref_type=ref_type,
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/listconfig.py", line 68, in __init__
    format_and_raise(node=None, key=None, value=None, cause=ex, msg=str(ex))
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/listconfig.py", line 66, in __init__
    self._set_value(value=content)
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/listconfig.py", line 521, in _set_value
    self.append(item)
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/listconfig.py", line 228, in append
    self._format_and_raise(key=index, value=item, cause=e)
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/_utils.py", line 694, in format_and_raise
    _raise(ex, cause)
  File "/root/anaconda3/envs/PiggybackGAN/lib/python3.6/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.UnsupportedValueType: Value 'Tensor' is not a supported primitive type
    full_key: netG_A_filter_list[0][0]
    reference_type=Optional[List[Any]]
    object_type=list

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Process finished with exit code 1

The information worth paying attention to is
 File "/opt/data/private/Pig/Piggyback-GAN-Pytorch-main/pb_cycleGAN.py", line 88, in train
    opt.netG_A_filter_list.append([layer.unc_filt.detach().cpu()])

omegaconf.errors.UnsupportedValueType: Value 'Tensor' is not a supported primitive type
    full_key: netG_A_filter_list[0][0]
    reference_type=Optional[List[Any]]
    object_type=list

My GPU only has a single core, not the 4 Gpus of the original author. I found the relevant part in the source code and conducted some tests to eliminate the problem of type mismatch. In addition, the version change of Omegaconf cannot solve my problem. On the right, the type of [layer.unc_filt.detach().cpu()] is list[tensor[]]
about code image
I don't know how to solve this problem now. I don't know whether to modify the code or be affected by the process. Could someone please tell me what I should do?

Output:
OmegaConf does not support assignment of non primitive types to the config. This have changed years ago.
There is a possibility that the author used a very old version of OmegaConf that did allow for this assignment, but based on his environment.yaml file he is using 2.0.6 which does not support it.
Contact the author of the code about this issue.


Instruction: How to handle hidden-cell output of 2-layer LSTM in PyTorch?

Input:
I have made a network with a LSTM and a fully connected layer in PyTorch. I want to test how an increase in the LSTM layers affects my performance.
Say my input is (6, 9, 14), meaning batch size 6, sequence size 9, and feature size 14, and I'm working on a task that has 6 classes, so I expect a 6-element one-hot-encoded tensor as the prediction for a single sequence. The output of this network after the FC layer should be (6, 6), however, if I use 2 LSTM layers it becomes (12, 6).
I don't understand how I should  handle the output of the LSTM layer to decrease the number of batches from [2 * batch_size] to [batch_size]. Also, I know I'm using the hidden state as the input to the FC layer, I want to try it this way for now.
Should I sum or concatenate every two batches or anything else?? Cheers!
    def forward(self, x):
        hidden_0 = torch.zeros((self.lstm_layers, x.size(0), self.hidden_size), dtype=torch.double, device=self.device)
        cell_0 = torch.zeros((self.lstm_layers, x.size(0), self.hidden_size), dtype=torch.double, device=self.device)

        y1, (hidden_1, cell_1) = self.lstm(x, (hidden_0, cell_0))
        hidden_1 = hidden_1.view(-1, self.hidden_size)

        y = self.linear(hidden_1)

        return y

Output:
The hidden state shape of a multi layer lstm is (layers, batch_size, hidden_size) see output LSTM. It contains the hidden state for each layer along the 0th dimension.
In your example you convert the shape into two dimensions here:
hidden_1 = hidden_1.view(-1, self.hidden_size)

this transforms the shape into (batch_size * layers, hidden_size).
What you would want to do is only use the hidden state of the last layer:
hidden = hidden_1[-1,:,:].view(-1, self.hidden_size)  # (1, bs, hidden) -> (bs, hidden)
y = self.linear(hidden)
return y


Instruction: TypeError: DataLoader found invalid type:

Input:
TypeError: DataLoader found invalid type: <class 'numpy.ndarray'>

Hi everyone, I have encountered difficulties, I can't find a solution, please help.
The program encountered an error at the train_fn () function.
train.py
from sklearn.preprocessing import StandardScaler
import joblib
from tqdm import tqdm
import pandas as pd
import numpy as np
import torch_geometric.transforms as T
import torch
import torch.optim as optim
# from torch_geometric.data import DataLoader
from torch_geometric.loader import DataLoader
from model import *
from Constant import *
import os
print(os.getcwd())
# path = '/home/ktcodes/jktModel/data/a09'
path = './data/a09'

e2e_emb = joblib.load(f'{path}/e2e_emb.pkl.zip')
c2c_emb = joblib.load(f'{path}/c2c_emb.pkl.zip')
skill_prob = joblib.load(f'{path}/skill_prob.pkl.zip')

filtered_skill_prob = {}
channel = 10
for i, skill_id in enumerate(skill_prob.index):
    if len(skill_prob[skill_id])>= channel:
        filtered_skill_prob[skill_id] = skill_prob[skill_id]
joblib.dump(filtered_skill_prob, f'{path}/filtered_skill_prob.pkl.zip')

# normalization
scaler = StandardScaler()
all_c_v = []
for k,v in c2c_emb.items():
    all_c_v.extend(list(v.numpy()))
all_c_v = scaler.fit_transform(np.array(all_c_v).reshape(-1,1))
all_c_v1 = {}
for i, (k,v) in enumerate(c2c_emb.items()):
    all_c_v1[k] = all_c_v[i*10:(i+1)*10].reshape(-1,)
all_e_v = {}
for skill,qu_embs in e2e_emb.items():
    q_num = qu_embs.shape[0]
    temp_all_v = qu_embs.numpy().reshape(-1,)
    temp_all_v = scaler.fit_transform(np.array(temp_all_v).reshape(-1,1))
    all_e_v[skill] = temp_all_v.reshape(-1,10)

skill_emb = {}
for skill in tqdm(filtered_skill_prob.keys()):
    temp_c = (np.array(all_c_v1[skill]))
    temp_e = np.array(np.mean(all_e_v[skill], axis=0))
    skill_emb[skill] = np.append(temp_c, temp_e)
prob_emb = {}
for skill in tqdm(filtered_skill_prob.keys()):
    for i, prob in enumerate(filtered_skill_prob[skill]):
        temp_c = (np.array(all_c_v1[skill]))
        temp_e = (np.array(all_e_v[skill][i]))
        new_emb = np.append(temp_c, temp_e)
        if prob in prob_emb.keys():
            prob_emb[prob] = np.row_stack((prob_emb[prob], new_emb)).squeeze().astype(np.int32)
#             print(prob_emb[prob].shape)
        else: prob_emb[prob] = new_emb
for prob in tqdm(prob_emb.keys()):
    if len(prob_emb[prob].shape) > 1:
        prob_emb[prob] = np.mean(prob_emb[prob], axis=0)

# Train/Test data
read_col = ['order_id', 'assignment_id', 'user_id', 'assistment_id', 'problem_id', 'correct',
            'sequence_id', 'base_sequence_id', 'skill_id', 'skill_name', 'original']
target = 'correct'
# read in the data
df = pd.read_csv(f'{path}/skill_builder_data.csv', low_memory=False, encoding="ISO-8859-1")[read_col]
df = df.sort_values(['order_id', 'user_id'])
# delete empty skill_id
df = df.dropna(subset=['skill_id'])
df = df[~df['skill_id'].isin(['noskill'])]
df.skill_id = df.skill_id.astype('int')
print('After removing empty skill_id, records number %d' % len(df))

# delete scaffolding problems
df = df[df['original'].isin([1])]
print('After removing scaffolding problems, records number %d' % len(df))

#delete the users whose interaction number is less than min_inter_num
min_inter_num = 3
users = df.groupby(['user_id'], as_index=True)
delete_users = []
for u in users:
    if len(u[1]) < min_inter_num:
        delete_users.append(u[0])
print('deleted user number based min-inters %d' % len(delete_users))
df = df[~df['user_id'].isin(delete_users)]
df = df[['user_id', 'problem_id', 'skill_id', 'correct']]
print('After deleting some users, records number %d' % len(df))
# print('features: ', df['assistment_id'].unique(), df['answer_type'].unique())

df = df[df['skill_id'].isin(filtered_skill_prob.keys())]
df['skill_cat'] = df['skill_id'].astype('category').cat.codes
df['e_emb'] = df['problem_id'].apply(lambda r: prob_emb[r])
df['c_emb'] = df['skill_id'].apply(lambda r: skill_emb[r])

group_c = df[['user_id', 'c_emb', 'correct']].groupby('user_id').apply(lambda r: (np.array(r['c_emb'].tolist()).squeeze(), r['correct'].values))
train_group_c = group_c.sample(frac=0.8, random_state=2020)
test_group_c = group_c[~group_c.index.isin(train_group_c.index)]
joblib.dump(train_group_c, f'{path}/train_group_c.pkl.zip')
joblib.dump(test_group_c, f'{path}/test_group_c.pkl.zip')

# print(type(train_group_c))
# # print(train_group_c.values)
# userid = train_group_c.index
# print(userid)
# q, qa = train_group_c[userid[0]]
# print(q, qa)

train_dataset = DKTDataset(train_group_c, max_seq=MAX_SEQ)
train_dataloader  = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)
valid_dataset = DKTDataset(test_group_c, max_seq=MAX_SEQ)
valid_dataloader  = DataLoader(valid_dataset, batch_size=BATCH_SIZE, shuffle=True)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = DKT(input_dim, hidden_dim, layer_dim, output_dim, device)
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
criterion = nn.BCEWithLogitsLoss()
scheduler = torch.optim.lr_scheduler.OneCycleLR(
    optimizer, max_lr=MAX_LEARNING_RATE, steps_per_epoch=len(train_dataloader), epochs=EPOCHS
)

model.to(device)
criterion.to(device)

for epoch in (range(EPOCHS)):
     # there
    loss, acc, auc = train_fn(model, train_dataloader, optimizer, criterion, device)
#     print("epoch - {}/{} train: - {:.3f} acc - {:.3f} auc - {:.3f}".format(epoch+1, EPOCHS, loss, acc, auc))
    loss, acc, pre, rec, f1, auc = valid_fn(model, valid_dataloader, criterion, device)

    res = "epoch - {}/{} valid: - {:.3f} acc - {:.3f} pre - {:.3f} rec - {:.3f} f1 - {:3f} auc - {:.3f}".format(epoch+1, EPOCHS, loss, acc, pre, rec, f1, auc)
    print(res)

The program does not go to this function：
def train_fn(model, dataloader, optimizer, criterion, scheduler=None,  device="cpu"):
    print('enter...')
    print("dataloader", type(dataloader))
    model.train()
    train_loss = []
    num_corrects = 0
    num_total = 0
    labels = []
    outs = []

    for x_emb, q_next, y in (dataloader):
        x = x_emb.to(device).float()
        y = y.to(device).float()
        q_next = q_next.to(device).float()

        out = model(x, q_next).squeeze().astype(np.int32)#[:, :-1]

        loss = criterion(out, y)
        loss.backward()
        optimizer.step()
#         scheduler.step()
        train_loss.append(loss.item())

        target_mask = (q_next!=0).unique(dim=2).squeeze().astype(np.int32)
#         target_mask = (y!=-1)

        filtered_out = torch.masked_select(out, target_mask)
        filtered_label = torch.masked_select(y, target_mask)
        filtered_pred = (torch.sigmoid(filtered_out) >= 0.5).long()

        num_corrects = num_corrects + (filtered_pred == filtered_label).sum().item()
        num_total = num_total + len(filtered_label)

        labels.extend(filtered_label.view(-1).data.cpu().numpy())
        outs.extend(filtered_pred.view(-1).data.cpu().numpy())

    acc = num_corrects / num_total
    auc = roc_auc_score(labels, outs)
    loss = np.mean(train_loss)

    return loss, acc, auc

Error info:
TypeError                                 Traceback (most recent call last)
~/kt/jktModel/embedding_dkt.py in <module>
    145 for epoch in (range(EPOCHS)):
    146     print("ashkdgjggvnskaj")
--> 147     loss, acc, auc = train_fn(model, train_dataloader, optimizer, criterion, device)
    148 #     print("epoch - {}/{} train: - {:.3f} acc - {:.3f} auc - {:.3f}".format(epoch+1, EPOCHS, loss, acc, auc))
    149     loss, acc, pre, rec, f1, auc = valid_fn(model, valid_dataloader, criterion, device)

~/kt/jktModel/model.py in train_fn(model, dataloader, optimizer, criterion, scheduler, device)
    110     model.train()
    111     train_loss = []
--> 112     num_corrects = 0
    113     num_total = 0
    114     labels = []

~/anaconda3/envs/dkt/lib/python3.8/site-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/envs/dkt/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    559     def _next_data(self):
    560         index = self._next_index()  # may raise StopIteration
--> 561         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    562         if self._pin_memory:
    563             data = _utils.pin_memory.pin_memory(data)

~/anaconda3/envs/dkt/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     50         else:
     51             data = self.dataset[possibly_batched_index]
---> 52         return self.collate_fn(data)

~/anaconda3/envs/dkt/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py in __call__(self, batch)
     32             return type(elem)(*(self(s) for s in zip(*batch)))
     33         elif isinstance(elem, Sequence) and not isinstance(elem, str):
---> 34             return [self(s) for s in zip(*batch)]
     35
     36         raise TypeError(f'DataLoader found invalid type: {type(elem)}')

~/anaconda3/envs/dkt/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py in <listcomp>(.0)
     32             return type(elem)(*(self(s) for s in zip(*batch)))
     33         elif isinstance(elem, Sequence) and not isinstance(elem, str):
---> 34             return [self(s) for s in zip(*batch)]
     35
     36         raise TypeError(f'DataLoader found invalid type: {type(elem)}')

~/anaconda3/envs/dkt/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py in __call__(self, batch)
     34             return [self(s) for s in zip(*batch)]
     35
---> 36         raise TypeError(f'DataLoader found invalid type: {type(elem)}')
     37
     38     def collate(self, batch):  # Deprecated...

TypeError: DataLoader found invalid type: <class 'numpy.ndarray'>

I have no ideas. This has already plagued me for a few days.

Output:
Slightly late, but in case anyone else encounters a similar issue - torch_geometric.loader.DataLoader cannot handle numpy arrays, so you must convert them to torch.Tensor in your dataset first. Alternatively, you could pass a custom collate function to the dataloader.
In this particular case, your DKTDataset returns tuples x_emb, q_next, y, some of which are of type numpy.ndarray, so convert them to tensors in DKTDataset.__get__().


Instruction: How to use gather() in python to return values at specific indices of a tensor

Input:
I have a tensor which looks like this:
tensor([[-0.0150,  0.1234],
    [-0.0184,  0.1062],
    [-0.0139,  0.1113],
    [-0.0088,  0.0726]])

And another that looks like this:
tensor([[1.],
    [1.],
    [0.],
    [0.]])

I want to return the values from the first tensor, for each row, that corresponds to the indice from the second tensor.
So our output would be:
tensor([0.1234], [0.1062], [-0.0139], [-0.0088]])

So far I have this code:
return torch.gather(tensor1, tensor2)

However I am getting the error:
TypeError: gather() received an invalid combination of arguments - got (Tensor, Tensor), but expected one of:
 * (Tensor input, int dim, Tensor index, *, bool sparse_grad, Tensor out)
 * (Tensor input, name dim, Tensor index, *, bool sparse_grad, Tensor out)

What am I doing wrong?

Output:
You are missing the dim argument.
You can see an example here: https://pytorch.org/docs/stable/generated/torch.gather.html
For your case I think that return torch.gather(tensor1, 1, tensor2) should work


Instruction: Fast GPU computation on PyTorch sparse tensor

Input:
Is it possible to do operations on each row of a PyTorch MxN tensor, but only at certain indices (for instance nonzero) to save time?
I'm particularly interested in the case of M and N very large where only a few elements on each row aren't null.
(Toy example) From this large tensor:
Large = Tensor([[0, 1, 3, 0, 0, 0],
                [0, 0, 0, 0, 5, 0],
                [1, 0, 0, 5, 0, 1]])

I'd like to use something like the following smaller "tensor":
irregular_tensor = [ [1, 3],
                     [5],
                     [1, 5, 1]]

and do the same exact computation on each row (for instance involving torch.cumsum and torch.exp) to obtain an output of size Mx1.
Is there a way to do that?

Output:
You might be interested in the Torch Sparse functionality. You can convert a PyTorch Tensor to a PyTorch Sparse tensor using the to_sparse() method of the Tensor class.
You can then access a tensor that contains all the indices in Coordinate format by the Sparse Tensor's indices() method, and a tensor that contains the associated values by the Sparse Tensor's values() method.
This also has the benefit of saving you memory in terms of storing the tensor.
There is some functionality for using other Torch functions on Sparse Tensors, but this is limited.
Also: be aware that this part of the API is still in Beta and subject to changes.


Instruction: Convert nn.Linear to nn.Conv1d

Input:
The format I want to output my model to doesn't support nn.Linear, so I'd like to change it to do the exact same thing but with nn.Conv1d.
My input is of shape (N, A, B) and I'd like to have a linear layer that transforms that into an output of shape (N, A, C). Previously, I was doing this with the layer nn.Linear(B, C). I'm able to produce working code that has the correct dimensions by doing
t1 = t1.transpose(1,2)
conv = nn.Conv1d(
            in_channels=B,
            out_channels=C,
            kernel_size=1
        )
t2 = conv(t1)
t2 = t2.transpose(1,2)

Is this functionally equivalent to doing t2 = nn.Linear(B,C)(t1)?
If so, is there a better/less verbose way of doing it?

Output:
Yes this is essentially doing the same thing.
Instead of transposing you could just add a trailing dummy dimension by doing
t1 = t1.unsqueeze(-1)
...
t2 = t2.squeeze(-1)

This has the advantage that the data doesn't have to be reordered, but the effect is probably negligible.


Instruction: open and read PT. file using python code by pytorch

Input:
I want to read a PT file with python and I don't know how, I want to open it with python
can you help me please, any ideas?

Output:
If your .PT file is related to the weight and bias of a model.
You must first install pytorch in your pc.
for more information  install go to this
then use this  :
model = torch.load(PATH)

saving_loading_models
You could iterate the parameters to get all weight and bias params via:(see weitht and bias)
for param in model.parameters():
   ....
# or
for name, param in model.named_parameters():
   ...


Instruction: How to evaluate a trained model in pytorch?

Input:
I have trained a model and save model using torch.save. Then after training I have loaded the model using train.load but I am getting this error

Traceback (most recent call last):
  File "/home/fsdfs.py", line 219, in <module>
    test(model, 'cuda', testloader)
  File "/home/fsdfs.py", line 201, in test
    model.eval()
AttributeError: 'collections.OrderedDict' object has no attribute 'eval'

Here is my code for test part
model = torch.load("train_5.pth")

def test(model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to('cuda'), target.to('cuda')
            output = model(data)
            #test_loss += f.cross_entropy(output, target, reduction='sum').item() # sum up batch loss
            pred = output.argmax(1, keepdim=True) # get the index of the max log-probability
            print(pred, target)
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print('\nTest set: Accuracy: {}/{} ({:.0f}%)\n'.format(
         correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

test(model, 'cuda', testloader)

I have commented training part of the code in the file, so in a way this and loading the data part is all that is there in the file now.
What am I doing wrong?

Output:
Like @jodag has said. you probably have saved a state_dict instead of a model, which is recommended by the community as well.
This link explains the difference between two. To keep my answer self contained, I copy the snippet from the documentation. Here is the recommended way:
Save:
torch.save(model.state_dict(), PATH)

Load:
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()

You could also save the entire model instead of saving the state_dict, if you really need to use the model the way you do.
Save:
torch.save(model, PATH)

Load:
# Model class must be defined somewhere
model = torch.load(PATH)
model.eval()


Instruction: For pytorch RNN model, can we inference the input by its output results?

Input:
all,
I wonder, torch.nn.rnn(~). If I know the final output, does it possible, I could inference its input value? For example:
myrnn = nn.RNN(4, 2, 1, batch_first=True)
expected_out, hidden = myrnn(input)
expected_out:  tensor([[[-0.7773, -0.2031]],

    [[-0.4129, -0.1802]],

    [[ 0.0599, -0.0151]],

    [[-0.9273,  0.2683]],

    [[ 0.6161,  0.5412]]])

Thank you so mcuh!!!

Output:
What you are asking is theoretically impossible
Neural networks in general represent functions that are impossible to inverse as they are not guaranteed to be byjective regardless of the underlying architecture.
This means that neither rnn nor any other neural network are invertible.


Instruction: Temporal Fusion Transformer (Pytorch Forecasting): `hidden_size` parameter

Input:
The Temporal-Fusion-Transformer (TFT) model in the PytorchForecasting package has several parameters (see: https://pytorch-forecasting.readthedocs.io/en/latest/_modules/pytorch_forecasting/models/temporal_fusion_transformer.html#TemporalFusionTransformer).
What does the hidden_size parameter exactly refer to? My best guess is that it refers  to the number of neurons contained in the GRN component of the TFT. If so, in which layer are these neurons contained?
I found the documentation not really helpful in this case, since they describe the hidden_size parameter as: "hidden size of network which is its main hyperparameter and can range from 8 to 512"
Side note: part of my ignorance might be due to the fact that I am not fully familiar with the individual components of the TFT model.

Output:
After a bit of research on the source code provided in the link, I was able to figure out how hidden_size is the main hyperparameter of the model. Here it is:
hidden_size describes indeed the number of neurons of each Dense layer of the GRN. You can check out the structure of the GRN at https://arxiv.org/pdf/1912.09363.pdf (page 6, Figure 2). Note that since the final layer of the GRN is just a normalization layer, also the output of the GRN has dimension hidden_size.
How is this the main hyperparameter of the model? By looking at the structure of the TFT model (on page 6 as well), the GRN unit appears in the Variable Selection process, in the Static Enrichment section and in the Position-wise Feed Forward section, so basically in every step of the learning process. Each one of these GRNs is built in the same way (only the input size varies).


Instruction: How to correctly combine LSTM with Linear layer

Input:
I got and LSTM that gives me output (4,32,32) i pass it to the Linear Layer(hidden size of LSTM, num_classes=1) and it gives me an output shape (4,32,1). I am trying to solve a wake word model for my AI assistant.
I have 2 classes i want to predict from. 0 is not wake up and 1 is the wake up AI.
My batch size is 32. But the output is (4,32,1). Isnt it should be 32,1 or something like that so i will know that there is one prediction for 1 audio mfcc?

Output: Not quite. You need to reshape your data to (32, 1) or (1, 32) in order for your linear layer to work. You can achieve this by adding a dimension with torch.unsqueeze() or even directly with torch.view(). If you use the unsqueeze function, the new shape should be (32, 1). If you use the view function, the new shape should be (1, 32).


Instruction: Pytorch dist.all_gather_object hangs

Input:
I'm using dist.all_gather_object (PyTorch version 1.8) to collect sample ids from all GPUs:
for batch in dataloader:
    video_sns = batch["video_ids"]
    logits = model(batch)
    group_gather_vdnames = [None for _ in range(envs['nGPU'])]
    group_gather_logits = [torch.zeros_like(logits) for _ in range(envs['nGPU'])]
    dist.all_gather(group_gather_logits, logits)
    dist.all_gather_object(group_gather_vdnames, video_sns)

The line dist.all_gather(group_gather_logits, logits) works properly,
but program hangs at line dist.all_gather_object(group_gather_vdnames, video_sns).
I wonder why the program hangs at dist.all_gather_object(), how can I fix it ?
EXTRA INFO:
I run my ddp code on a local machine with multiple GPUs. The start script is:
export NUM_NODES=1
export NUM_GPUS_PER_NODE=2
export NODE_RANK=0
export WORLD_SIZE=$(($NUM_NODES * $NUM_GPUS_PER_NODE))

python -m torch.distributed.launch \
       --nproc_per_node=$NUM_GPUS_PER_NODE \
       --nnodes=$NUM_NODES \
       --node_rank $NODE_RANK \
       main.py \
       --my_args

Output:
Turns out we need to set the device id manually as mentioned in the docstring of dist.all_gather_object() API.
Adding
torch.cuda.set_device(envs['LRANK']) # my local gpu_id

and the codes work.
I always thought the GPU ID is set automatically by PyTorch dist, turns out it's not.


Instruction: Inspect signature of a python function without the __code__ attribute (e.g. PyTorch)

Input:
I am trying to determine the signature of a PyTorch function at runtime (e.g. torch.empty or torch.zeros). But something like inspect.signature(torch.empty) doesn't work here:
>>> import inspect
>>> import torch
>>> def add(a,b):
...     return a+b
...
>>> inspect.signature(add)
<Signature (a, b)>
>>> inspect.signature(torch.empty)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/anaconda/envs/gc/lib/python3.8/inspect.py", line 3105, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
  File "/home/anaconda/envs/gc/lib/python3.8/inspect.py", line 2854, in from_callable
    return _signature_from_callable(obj, sigcls=cls,
  File "/home/anaconda/envs/gc/lib/python3.8/inspect.py", line 2308, in _signature_from_callable
    return _signature_from_builtin(sigcls, obj,
  File "/home/anaconda/envs/gc/lib/python3.8/inspect.py", line 2119, in _signature_from_builtin
    raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in method empty of type object at 0x7f382e1321c0>

I am guessing the underlying reason to be the absence of the __code__ attribute
>>> add.__code__
<code object add at 0x7f382f3a9710, file "<stdin>", line 1>
>>> torch.empty.__code__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'builtin_function_or_method' object has no attribute '__code__'

Is there any way to inspect the signatures of python functions in such cases?

Output:
May be not best option, but workaround, - parse torch.empty.__doc__
print(torch.empty.__doc__)

    empty(*size, *, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False, pin_memory=False, memory_format=torch.contiguous_format) -> Tensor

Returns a tensor filled with uninitialized data. The shape of the tensor is
defined by the variable argument :attr:`size`.

Args:
...


Instruction: Pytorch AttributeError: can't set attribute

Input:
I'm using pytorch lightining and I have this error but I'm non really understanding what is the problem. I create a Deep Learning pipeline to run with hyperparameters searching and I think that the problem is in.
I omitted some part of the code because I think they are irrelevant for this issue (due to stackoverflow restrictions). Thanks for the help!
class ProtBertBFDClassifier(pl.LightningModule):

    def __init__(self,hparams) -> None:
        super(ProtBertBFDClassifier, self).__init__()
        self.hparams = hparams
        self.batch_size = self.hparams.batch_size

        self.model_name = pretrained_model_name

        self.dataset = Loc_dataset()

        self.metric_acc = Accuracy()

        # build model
        self.__build_model()

        # Loss criterion initialization.
        self.__build_loss()

        if self.hparams.nr_frozen_epochs > 0:
            self.freeze_encoder()
        else:
            self._frozen = False
        self.nr_frozen_epochs = self.hparams.nr_frozen_epochs

    def __build_model(self) -> None:
        """ Init BERT model + tokenizer + classification head."""
        self.ProtBertBFD = BertModel.from_pretrained(self.model_name,gradient_checkpointing=self.hparams.gradient_checkpointing)
        self.encoder_features = 1024

        # Tokenizer
        self.tokenizer = BertTokenizer.from_pretrained(self.model_name, do_lower_case=False)

        # Label Encoder
        self.label_encoder = LabelEncoder(
            self.hparams.label_set.split(","), reserved_labels=[]
        )
        self.label_encoder.unknown_index = None

        # Classification head
        self.classification_head = nn.Sequential(
            nn.Linear(self.encoder_features*4, self.label_encoder.vocab_size),
            nn.Tanh(),
        )

    .....
    def predict(self, sample: dict) -> dict:
        """ Predict function.
        :param sample: dictionary with the text we want to classify.
        Returns:
            Dictionary with the input text and the predicted label.
        """
......

    def pool_strategy(self, features,
                      pool_cls=True, pool_max=True, pool_mean=True,
                      pool_mean_sqrt=True):
        token_embeddings = features['token_embeddings']
        cls_token = features['cls_token_embeddings']
        attention_mask = features['attention_mask']

        ## Pooling strategy
        output_vectors = []
        if pool_cls:
            output_vectors.append(cls_token)
        if pool_max:
            input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
            token_embeddings[input_mask_expanded == 0] = -1e9  # Set padding tokens to large negative value
            max_over_time = torch.max(token_embeddings, 1)[0]
            output_vectors.append(max_over_time)
        if pool_mean or pool_mean_sqrt:
            input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
            sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)

            #If tokens are weighted (by WordWeights layer), feature 'token_weights_sum' will be present
            if 'token_weights_sum' in features:
                sum_mask = features['token_weights_sum'].unsqueeze(-1).expand(sum_embeddings.size())
            else:
                sum_mask = input_mask_expanded.sum(1)

            sum_mask = torch.clamp(sum_mask, min=1e-9)

            if pool_mean:
                output_vectors.append(sum_embeddings / sum_mask)
            if pool_mean_sqrt:
                output_vectors.append(sum_embeddings / torch.sqrt(sum_mask))

        output_vector = torch.cat(output_vectors, 1)
        return output_vector

   ........

        inputs = self.tokenizer.batch_encode_plus(sample["seq"],
                                                  add_special_tokens=True,
                                                  padding=True,
                                                  truncation=True,
                                                  max_length=self.hparams.max_length)

        if not prepare_target:
            return inputs, {}

        # Prepare target:
        try:
            targets = {"labels": self.label_encoder.batch_encode(sample["label"])}
            return inputs, targets
        except RuntimeError:
            print(sample["label"])
            raise Exception("Label encoder found an unknown label.")

......

    def validation_step(self, batch: tuple, batch_nb: int, *args, **kwargs) -> dict:
        """ Similar to the training step but with the model in eval mode.
        Returns:
            - dictionary passed to the validation_end function.
        """
        inputs, targets = batch

        model_out = self.forward(**inputs)
        loss_val = self.loss(model_out, targets)

        y = targets["labels"]
        y_hat = model_out["logits"]

        labels_hat = torch.argmax(y_hat, dim=1)
        val_acc = self.metric_acc(labels_hat, y)

        output = OrderedDict({"val_loss": loss_val, "val_acc": val_acc,})

        return output

    def validation_epoch_end(self, outputs: list) -> dict:
        """ Function that takes as input a list of dictionaries returned by the validation_step
        function and measures the model performance accross the entire validation set.

        Returns:
            - Dictionary with metrics to be added to the lightning logger.
        """

        val_loss_mean = torch.stack([x['val_loss'] for x in outputs]).mean()
        val_acc_mean = torch.stack([x['val_acc'] for x in outputs]).mean()

        tqdm_dict = {"val_loss": val_loss_mean, "val_acc": val_acc_mean}
        result = {
            "progress_bar": tqdm_dict,
            "log": tqdm_dict,
            "val_loss": val_loss_mean,
        }
        return result

.......

    def test_epoch_end(self, outputs: list) -> dict:
        """ Function that takes as input a list of dictionaries returned by the validation_step
        function and measures the model performance accross the entire validation set.

        Returns:
            - Dictionary with metrics to be added to the lightning logger.
        """
        test_loss_mean = torch.stack([x['test_loss'] for x in outputs]).mean()
        test_acc_mean = torch.stack([x['test_acc'] for x in outputs]).mean()

        tqdm_dict = {"test_loss": test_loss_mean, "test_acc": test_acc_mean}
        result = {
            "progress_bar": tqdm_dict,
            "log": tqdm_dict,
            "test_loss": test_loss_mean,
        }
        return result

    def configure_optimizers(self):
        """ Sets different Learning rates for different parameter groups. """
        parameters = [
            {"params": self.classification_head.parameters()},
            {
                "params": self.ProtBertBFD.parameters(),
                "lr": self.hparams.encoder_learning_rate,
            },
        ]
        optimizer = optim.Adam(parameters, lr=self.hparams.learning_rate)
        return [optimizer], []

    def __retrieve_dataset(self, train=True, val=True, test=True):
        """ Retrieves task specific dataset """
        if train:
            return self.dataset.load_dataset(hparams.train_csv)
        elif val:
            return self.dataset.load_dataset(hparams.dev_csv)
        elif test:
            return self.dataset.load_dataset(hparams.test_csv)
        else:
            print('Incorrect dataset split')

    def train_dataloader(self) -> DataLoader:
        """ Function that loads the train set. """
        self._train_dataset = self.__retrieve_dataset(val=False, test=False)
        return DataLoader(
            dataset=self._train_dataset,
            sampler=RandomSampler(self._train_dataset),
            batch_size=self.hparams.batch_size,
            collate_fn=self.prepare_sample,
            num_workers=self.hparams.loader_workers,
        )

....

    @classmethod
    def add_model_specific_args(
        cls, parser: HyperOptArgumentParser
    ) -> HyperOptArgumentParser:
        """ Parser for Estimator specific arguments/hyperparameters.
        :param parser: HyperOptArgumentParser obj
        Returns:
            - updated parser
        """
        parser.opt_list(
            "--max_length",
            default=1536,
            type=int,
            help="Maximum sequence length.",
        )
        parser.add_argument(
            "--encoder_learning_rate",
            default=5e-06,
            type=float,
            help="Encoder specific learning rate.",
        )

        return parser

# these are project-wide arguments
parser = HyperOptArgumentParser(
    strategy="random_search",
    description="Minimalist ProtBERT Classifier",
    add_help=True,
)

# Early Stopping
parser.add_argument(
    "--monitor", default="val_acc", type=str, help="Quantity to monitor."
)
parser.add_argument(
    "--metric_mode",
    default="max",
    type=str,
    help="If we want to min/max the monitored quantity.",
    choices=["auto", "min", "max"],
)
parser.add_argument(
    "--patience",
    default=5,
    type=int,
    help=(
        "Number of epochs with no improvement "
        "after which training will be stopped."
    ),
)

parser.add_argument(
    "--accumulate_grad_batches",
    default=32,
    type=int,
    help=(
        "Accumulated gradients runs K small batches of size N before "
        "doing a backwards pass."
    ),
)

# gpu/tpu args
parser.add_argument("--gpus", type=int, default=1, help="How many gpus")
parser.add_argument("--tpu_cores", type=int, default=None, help="How many tpus")
parser.add_argument(
    "--val_percent_check",
    default=1.0,
    type=float,
    help=(
        "If you don't want to use the entire dev set (for debugging or "
        "if it's huge), set how much of the dev set you want to use with this flag."
    ),
)

# each LightningModule defines arguments relevant to it
parser = ProtBertBFDClassifier.add_model_specific_args(parser)

hparams = parser.parse_known_args()[0]

"""
Main training routine specific for this project
:param hparams:
"""
seed_everything(hparams.seed)

# ------------------------
# 1 INIT LIGHTNING MODEL
# ------------------------
model = ProtBertBFDClassifier(hparams)

This is the error:

1 frames
<ipython-input-26-561494d91469> in __init__(self)
     10     def __init__(self) -> None:
     11         super(ProtBertBFDClassifier, self).__init__()
---> 12         self.hparams = parser.parse_known_args()[0]
     13         self.batch_size = self.hparams.batch_size
     14

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __setattr__(self, name, value)
   1223                     buffers[name] = value
   1224                 else:
-> 1225                     object.__setattr__(self, name, value)
   1226
   1227     def __delattr__(self, name):

AttributeError: can't set attribute

Output: pip install pytorch-lightning==1.2.10


Instruction: Find index where a sub-tensor does not equal to a given tensor in Pytorch

Input:
I have a tensor, for example,
a = [[15,30,0,2], [-1,-1,-1,-1], [10, 20, 40, 60], [-1,-1,-1,-1]]

which has the shape (4,4).
How can I find the index where a specific sub-tensor
[-1,-1,-1,-1]

that doesn't appear using PyTorch. The expected output I want to get is
[0,2]

Output:
You can compare the elements for each row of the tensor using torch.any(), and then use .nonzero() and .flatten() to generate the indices:
torch.any(a != torch.Tensor([-1, -1, -1, -1]), axis=1).nonzero().flatten()

For example,
import torch

a = torch.Tensor([[15,30,0,2], [-1,-1,-1,-1], [10, 20, 40, 60], [-1,-1,-1,-1]])
result = torch.any(a != torch.Tensor([-1, -1, -1, -1]), axis=1).nonzero().flatten()
print(result)

outputs:
tensor([0, 2])


Instruction: Reproducibility issue with PyTorch

Input:
I'm running a script with the same seed and I see results are reproduced on consecutive runs but somehow running the same script with the same seed changes the output after a few days. I'm only getting a short-term reproducibility which is weird. For reproducibility my script includes the following statements already:
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.use_deterministic_algorithms(True)

random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)

I also checked the sequence of instance ids created by the RandomSampler for train Dataloader which is maintained across runs. Also set the num_workers=0 in the dataloader.What could be causing the output to change?

Output:
PyTorch is actually not fully deterministic. Meaning that with a set seed, some PyTorch operations will simply behave differently and diverge from previous runs, given enough time. This is due to algorithm, CUDA, and backprop optimizations.
This is a good read: https://pytorch.org/docs/stable/notes/randomness.html
The above page lists which operations are non-deterministic. It is generally discouraged that you disable their use, but it can be done with:
torch.use_deterministic_algorithms()

This might also disable which operation can be used.


Instruction: ValueError: num_samples should be a positive integer value, but got num_samples=0

Input:
I have data organized as follows: /dataset/train_or_validation/neg_or_pos_class/images.png
So, inside train or validation I have 2 folders, 1 for negative and 1 for positive.
I have the error of the title ValueError: num_samples should be a positive integer value, but got num_samples=0 because basically I am inside /dataset/train_or_validation, but then I need to access the folders neg or pos. Images are in this format: MCUCXR_0000_1.png for positive images, while MCUCXR_0000_0.png for negative class. I was thinking to extract all the images from the folders, in order to have /dataset/train_or_validation/images.png, but in this case how I can specify which is the class?
Or, how can I iterate through the positive/negative folders?
This is my code:
"""Montgomery Shard Descriptor."""

import logging
import os
from typing import List
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from pathlib import Path

import numpy as np
import requests

from openfl.interface.interactive_api.shard_descriptor import ShardDataset
from openfl.interface.interactive_api.shard_descriptor import ShardDescriptor

from torchvision import transforms

# Compose transformations
train_transform = transforms.Compose([
  transforms.RandomHorizontalFlip(),
  transforms.RandomVerticalFlip(),
  transforms.Resize((512, 512)),
  transforms.ToTensor(),
])

test_transform = transforms.Compose([
  transforms.Resize((512, 512)),
  transforms.ToTensor(),
])

logger = logging.getLogger(__name__)

class MontgomeryShardDataset(ShardDataset):
    """Montgomery Shard dataset class."""

    def __init__(self, dataset_dir: Path, dataset_type: str,):
        """Initialize MontgomeryDataset."""
        self.data_type = dataset_type
        self.dataset_dir = dataset_dir
        print(self.dataset_dir)
        self.imgs_path = list(dataset_dir.glob('*.png'))

    def __getitem__(self, index: int):
        """Return an item by the index."""
        img_path = self.imgs_path[index]
        img = Image.open(img_path)
        return img

    def __len__(self):
        """Return the len of the dataset."""
        return len(self.imgs_path)

class MontgomeryShardDescriptor(ShardDescriptor):
    """Montgomery Shard descriptor class."""

    def __init__(
            self,
            data_folder: str = 'montgomery_data',
            **kwargs
    ):
        """Initialize MontgomeryShardDescriptor."""
        #print("Path at terminal when executing this file")
        print(os.getcwd() + "\n")
        #print(self.common_data_folder)
        self.data_folder = data_folder
        self.dataset_dir = Path.cwd() / data_folder
        trainset, testset = self.get_data()
        print("IO SONO" + "\n")
        print(self.dataset_dir)
        self.data_by_type = {
            'train': self.dataset_dir / 'TRAIN',
            'val': self.dataset_dir / 'TEST'
        }

    def get_shard_dataset_types(self) -> List[str]:
        """Get available shard dataset types."""
        return list(self.data_by_type)

    def get_dataset(self, dataset_type='train'):
        """Return a shard dataset by type."""
        print("Path at terminal when executing this file")
        print(os.getcwd() + "\n")
        #os.chdir("/home/lmancuso/openfl/openfl-tutorials/interactive_api/OPENLAB/envoy")
        if dataset_type not in self.data_by_type:
            raise Exception(f'Wrong dataset type: {dataset_type}')
        return MontgomeryShardDataset(
            dataset_dir=self.data_by_type[dataset_type],
            dataset_type=dataset_type,
        )

    @property
    def sample_shape(self):
        """Return the sample shape info."""
        return ['3', '512', '512']

    @property
    def target_shape(self):
        """Return the target shape info."""
        return ['3', '512', '512']

    @property
    def dataset_description(self) -> str:
        """Return the dataset description."""
        return (f'Montgomery dataset, shard number')

    def get_data(self):
        root_dir = "montgomery_data"
        #train_set = ImageFolder(os.path.join(root_dir, "TRAIN"), transform=train_transform)
        #test_set = ImageFolder(os.path.join(root_dir, "TEST"), transform=test_transform)
        train_set = os.path.join(root_dir, "TRAIN")
        test_set = os.path.join(root_dir, "TEST")

        print('Montgomery data was loaded!')
        return train_set, test_set

I am using the framework for Federated Learning developed by Intel, OpenFL.
As you can see I tried also to use ImageFolder because I think it can be useful in this case.
EDIT with the full traceback:
new_state[k] = pt.from_numpy(tensor_dict.pop(k)).to(device)
           ERROR    Collaborator failed with error: num_samples should be a positive integer value, but got num_samples=0:                           envoy.py:93
                    Traceback (most recent call last):
                      File "/home/lmancuso/openfl/openfl/component/envoy/envoy.py", line 91, in run
                        self._run_collaborator()
                      File "/home/lmancuso/openfl/openfl/component/envoy/envoy.py", line 164, in _run_collaborator
                        col.run()
                      File "/home/lmancuso/openfl/openfl/component/collaborator/collaborator.py", line 145, in run
                        self.do_task(task, round_number)
                      File "/home/lmancuso/openfl/openfl/component/collaborator/collaborator.py", line 259, in do_task
                        **kwargs)
                      File "/home/lmancuso/openfl/openfl/federated/task/task_runner.py", line 117, in collaborator_adapted_task
                        loader = self.data_loader.get_train_loader()
                      File "/tmp/ipykernel_8572/1777129341.py", line 35, in get_train_loader
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 262, in __init__
                        sampler = RandomSampler(dataset, generator=generator)  # type: ignore
                      File "/home/lmancuso/bruno/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 104, in __init__
                        "value, but got num_samples={}".format(self.num_samples))
                    ValueError: num_samples should be a positive integer value, but got num_samples=0
           INFO     Send WaitExperiment request                                                                                director_client.py:80
           INFO     WaitExperiment response has received                                                                                director_client.py:82

Output: The problem is that the dataset is empty. The datapath may be wrong or preprocessing might be causing problems ending up with no object in Dataset object.


Instruction: PytorchStreamReader failed reading zip archive: failed finding central directory

Input:
I am trying to learn pytorch from a book, but it seems not a straight line for me.
I coped the code below and pasted in my jupyter notebook for running but it gave me an error I am not able to interpret at my level!
from torchvision import models
model = models.alexnet(pretrained=True)

# set the device
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'Device: {device}')

model.eval()
model.to(device)
y = model(batch.to(device))
print(y.shape)

The error is as below
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-50-03488007067c> in <module>
      1 from torchvision import models
----> 2 model = models.alexnet(pretrained=True)
      3
      4 # set the device
      5 device = 'cuda' if torch.cuda.is_available() else 'cpu'

~\anaconda3\lib\site-packages\torchvision\models\alexnet.py in alexnet(pretrained, progress, **kwargs)
     61     model = AlexNet(**kwargs)
     62     if pretrained:
---> 63         state_dict = load_state_dict_from_url(model_urls['alexnet'],
     64                                               progress=progress)
     65         model.load_state_dict(state_dict)

~\anaconda3\lib\site-packages\torch\hub.py in load_state_dict_from_url(url, model_dir, map_location, progress, check_hash, file_name)
    555     if _is_legacy_zip_format(cached_file):
    556         return _legacy_zip_load(cached_file, model_dir, map_location)
--> 557     return torch.load(cached_file, map_location=map_location)

~\anaconda3\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    598             # reset back to the original position.
    599             orig_position = opened_file.tell()
--> 600             with _open_zipfile_reader(opened_file) as opened_zipfile:
    601                 if _is_torchscript_zip(opened_zipfile):
    602                     warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"

~\anaconda3\lib\site-packages\torch\serialization.py in __init__(self, name_or_buffer)
    240 class _open_zipfile_reader(_opener):
    241     def __init__(self, name_or_buffer) -> None:
--> 242         super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
    243
    244

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

someone help me understand this please.
Thank you.

Output: I think this issue happens when the file is not downloaded completely.


Instruction: Monai : RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 7 but got size 8 for tensor number 1 in the list

Input:
I am using Monai for the 3D Multilabel segmentation task. My input image size is 512x496x49 and my label size is 512x496x49. An Image can have 3 labels in one image. With transform, I have converted the image in size 1x512x512x49 and Label in 3x512x512x49
My Transform
# Setting tranform for train and test data
a_min=6732
a_max=18732

train_transform = Compose(
    [
      LoadImaged(keys=["image", "label"]),
      EnsureChannelFirstd(keys="image"),
      ConvertToMultiChannelBasedOnBratsClassesd(keys="label"),
      ScaleIntensityRanged(keys='image', a_min=a_min, a_max=a_max, b_min=0.0, b_max=1.0, clip=False),
      Orientationd(keys=["image", "label"], axcodes="RAS"),
      # Spacingd(keys=["image", "label"], pixdim=(
      #     1.5, 1.5, 2.0), mode=("bilinear", "nearest")),
      RandFlipd(keys=["image", "label"], prob=0.5, spatial_axis=0),
      RandFlipd(keys=["image", "label"], prob=0.5, spatial_axis=1),
      RandFlipd(keys=["image", "label"], prob=0.5, spatial_axis=2),
      CropForegroundd(keys=["image", "label"], source_key="image"),
      NormalizeIntensityd(keys="image", nonzero=True, channel_wise=True),
      SpatialPadd(keys=['image', 'label'], spatial_size= [512, 512, 49]),# it will result in 512x512x49
      EnsureTyped(keys=["image", "label"]),
    ]
)
val_transform = Compose(
    [
      LoadImaged(keys=["image", "label"]),
      EnsureChannelFirstd(keys="image"),
      ConvertToMultiChannelBasedOnBratsClassesd(keys="label"),
      ScaleIntensityRanged(keys='image', a_min=a_min, a_max=a_max, b_min=0.0, b_max=1.0, clip=False),
      Orientationd(keys=["image", "label"], axcodes="RAS"),
      # Spacingd(keys=["image", "label"], pixdim=(
      #     1.5, 1.5, 2.0), mode=("bilinear", "nearest")),
      CropForegroundd(keys=["image", "label"], source_key="image"),
      NormalizeIntensityd(keys="image", nonzero=True, channel_wise=True),
      SpatialPadd(keys=['image', 'label'], spatial_size= [512, 512, 49]),# it will result in 512x512x49
      EnsureTyped(keys=["image", "label"]),
    ]
)

Dataloader for training and val
train_ds = CacheDataset(data=train_files, transform=train_transform,cache_rate=1.0, num_workers=4)
train_loader = DataLoader(train_ds, batch_size=2, shuffle=True, num_workers=4,collate_fn=pad_list_data_collate)

val_ds = CacheDataset(data=val_files, transform=val_transform, cache_rate=1.0, num_workers=4)
val_loader = DataLoader(val_ds, batch_size=1, num_workers=4)

3D U-Net Network from Monai
# standard PyTorch program style: create UNet, DiceLoss and Adam optimizer

device = torch.device("cuda:0")
model = UNet(
    spatial_dims=3,
    in_channels=1,
    out_channels=4,
    channels=(16, 32, 64, 128, 256),
    strides=(2, 2, 2, 2),
    num_res_units=2,
    norm=Norm.BATCH,
).to(device)
loss_function = DiceLoss(to_onehot_y=True, sigmoid=True)
optimizer = torch.optim.Adam(model.parameters(), 1e-4)
dice_metric = DiceMetric(include_background=True, reduction="mean")

Training
max_epochs = 5
val_interval = 2
best_metric = -1
best_metric_epoch = -1
epoch_loss_values = []
metric_values = []
post_pred = Compose([EnsureType(), AsDiscrete(argmax=True, to_onehot=4)])
post_label = Compose([EnsureType(), AsDiscrete(to_onehot=4)])

for epoch in range(max_epochs):
    print("-" * 10)
    print(f"epoch {epoch + 1}/{max_epochs}")
    model.train()
    epoch_loss = 0
    step = 0
    for batch_data in train_loader:
        step += 1
        inputs, labels = (
            batch_data["image"].to(device),
            batch_data["label"].to(device),
        )
        optimizer.zero_grad()
        print("Size of inputs :", inputs.shape)
        print("Size of inputs[0] :", inputs[0].shape)
        # print("Size of inputs[1] :", inputs[1].shape)
        # print("printing of inputs :", inputs)
        outputs = model(inputs)
        loss = loss_function(outputs, labels)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        print(
            f"{step}/{len(train_ds) // train_loader.batch_size}, "
            f"train_loss: {loss.item():.4f}")
    epoch_loss /= step
    epoch_loss_values.append(epoch_loss)
    print(f"epoch {epoch + 1} average loss: {epoch_loss:.4f}")

    if (epoch + 1) % val_interval == 0:
        model.eval()
        with torch.no_grad():
            for val_data in val_loader:
                val_inputs, val_labels = (
                    val_data["image"].to(device),
                    val_data["label"].to(device),
                )
                roi_size = (160, 160, 160)
                sw_batch_size = 4
                val_outputs = sliding_window_inference(
                    val_inputs, roi_size, sw_batch_size, model)
                val_outputs = [post_pred(i) for i in decollate_batch(val_outputs)]
                val_labels = [post_label(i) for i in decollate_batch(val_labels)]
                # compute metric for current iteration
                dice_metric(y_pred=val_outputs, y=val_labels)

            # aggregate the final mean dice result
            metric = dice_metric.aggregate().item()
            # reset the status for next validation round
            dice_metric.reset()

            metric_values.append(metric)
            if metric > best_metric:
                best_metric = metric
                best_metric_epoch = epoch + 1
                torch.save(model.state_dict(), os.path.join(
                    root_dir, "best_metric_model.pth"))
                print("saved new best metric model")
            print(
                f"current epoch: {epoch + 1} current mean dice: {metric:.4f}"
                f"\nbest mean dice: {best_metric:.4f} "
                f"at epoch: {best_metric_epoch}"
            )

While training I am getting this error
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 7 but got size 8 for tensor number 1 in the list.

I followed the 3D Segmentation Monai tutorial but this was only for 2 classes (including background) therefore I followed the discussion at https://github.com/Project-MONAI/MONAI/issues/415 but even though I changed what was recommended in this discussion still am getting errors while training.

Output: Your images have a depth of 49, but due to the 4 downsampling steps, each with stride 2, your images need to be divisible by a factor of 2**4=16. Adding in DivisiblePadd(["image", "label"], 16) should solve it.


Instruction: Deep Smote error : RuntimeError: mat1 and mat2 shapes cannot be multiplied (51200x1 and 512x300)

Input:
I am trying to run deep Smote on cifar10 and Dont have much experience with pytorch as I code in tensorflow. It works fine when I run it on MNIST and FMNIST keeping channles = 1 there
However, the moment i try it on cifar10, i dosent behave well.
The code given in the paper says that it works for Cifar10 too,
All the help is appreciated
Here is the link to the source code of the paper
https://github.com/dd1github/DeepSMOTE
The source code is in Tensorflow, can someone pls help me here
RuntimeError                              Traceback (most recent call last)
C:\Users\RESEAR~1\AppData\Local\Temp/ipykernel_24844/1514724550.py in <module>
     93
     94                 # run images
---> 95                 z_hat = encoder(images)
     96
     97                 x_hat = decoder(z_hat) #decoder outputs tanh

~\.conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

~\.conda\envs\pytorch\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1846     if has_torch_function_variadic(input, weight, bias):
   1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
   1849
   1850

RuntimeError: mat1 and mat2 shapes cannot be multiplied (51200x1 and 512x300)

Here's the code :
## create encoder model and decoder model
class Encoder(nn.Module):
    def __init__(self, args):
        super(Encoder, self).__init__()

        self.n_channel = args['n_channel']
        self.dim_h = args['dim_h']
        self.n_z = args['n_z']

        # convolutional filters, work excellent with image data
        self.conv = nn.Sequential(
            nn.Conv2d(self.n_channel, self.dim_h, 4, 2, 1, bias=False),
            #nn.ReLU(True),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(self.dim_h, self.dim_h * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(self.dim_h * 2),
            #nn.ReLU(True),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(self.dim_h * 2, self.dim_h * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(self.dim_h * 4),
            #nn.ReLU(True),
            nn.LeakyReLU(0.2, inplace=True),

#             nn.Conv2d(self.dim_h * 4, self.dim_h * 8, 4, 2, 1, bias=False),

            #3d and 32 by 32
            nn.Conv2d(self.dim_h * 4, self.dim_h * 8, 4, 1, 0, bias=False),

            nn.BatchNorm2d(self.dim_h * 8), # 40 X 8 = 320
            #nn.ReLU(True),
            nn.LeakyReLU(0.2, inplace=True) )#,
            #nn.Conv2d(self.dim_h * 8, 1, 2, 1, 0, bias=False))
            #nn.Conv2d(self.dim_h * 8, 1, 4, 1, 0, bias=False))
        # final layer is fully connected
        print("linearer >>>>>>>> ",self.dim_h * (2 ** 3), self.n_z)
        self.fc = nn.Linear(self.dim_h * (2 ** 3), self.n_z)
        print("leeeeeeeeeee ")

    def forward(self, x):
        #print('enc')
        #print('input ',x.size()) #torch.Size([100, 3,32,32])
        x = self.conv(x)

#         x = x.squeeze()
#         print('aft squeeze ',x.size()) #torch.Size([128, 320])
        #aft squeeze  torch.Size([100, 320])
        x = self.fc(x)
        #print('out ',x.size()) #torch.Size([128, 20])
        #out  torch.Size([100, 300])
        return x

class Decoder(nn.Module):
    def __init__(self, args):
        super(Decoder, self).__init__()

        self.n_channel = args['n_channel']
        self.dim_h = args['dim_h']
        self.n_z = args['n_z']

        # first layer is fully connected
        self.fc = nn.Sequential(
            nn.Linear(self.n_z, self.dim_h * 8 * 7 * 7),
            nn.ReLU())

        # deconvolutional filters, essentially inverse of convolutional filters
        self.deconv = nn.Sequential(
            nn.ConvTranspose2d(self.dim_h * 8, self.dim_h * 4, 4),
            nn.BatchNorm2d(self.dim_h * 4),
            nn.ReLU(True),
            nn.ConvTranspose2d(self.dim_h * 4, self.dim_h * 2, 4),
            nn.BatchNorm2d(self.dim_h * 2),
            nn.ReLU(True),
            nn.ConvTranspose2d(self.dim_h * 2, 1, 4, stride=2),
            #nn.Sigmoid())
            nn.Tanh())

    def forward(self, x):
        #print('dec')
        #print('input ',x.size())
        x = self.fc(x)
        x = x.view(-1, self.dim_h * 8, 7, 7)
        x = self.deconv(x)
        return x

..........
#NOTE: Download the training ('.../0_trn_img.txt') and label files
# ('.../0_trn_lab.txt').  Place the files in directories (e.g., ../MNIST/trn_img/
# and /MNIST/trn_lab/).  Originally, when the code was written, it was for 5 fold
#cross validation and hence there were 5 files in each of the
#directories.  Here, for illustration, we use only 1 training and 1 label
#file (e.g., '.../0_trn_img.txt' and '.../0_trn_lab.txt').
path = "C:/Users/antpc/Documents/saqib_smote/fmnist/"
path = "C:/Users/Research6/Desktop/smote experimentation/mnist/"
dtrnimg = (path+'/CBL_images')
dtrnlab = (path+'/CBL_labels')

ids = os.listdir(dtrnimg)
idtri_f = [os.path.join(dtrnimg, image_id) for image_id in ids]
print(idtri_f)

ids = os.listdir(dtrnlab)
idtrl_f = [os.path.join(dtrnlab, image_id) for image_id in ids]
print(idtrl_f)

#for i in range(5):
for i in range(len(ids)):
    print()
    print(i)
    encoder = Encoder(args)
    decoder = Decoder(args)

    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    print(device)
    decoder = decoder.to(device)
    encoder = encoder.to(device)

    train_on_gpu = torch.cuda.is_available()

    #decoder loss function
    criterion = nn.MSELoss()
    criterion = criterion.to(device)

    trnimgfile = idtri_f[i]
    trnlabfile = idtrl_f[i]

    print(trnimgfile)
    print(trnlabfile)
    dec_x = np.loadtxt(trnimgfile)
    dec_y = np.loadtxt(trnlabfile)

    print('train imgs before reshape ',dec_x.shape)
    print('train labels ',dec_y.shape)
    print(collections.Counter(dec_y))
#     dec_x = dec_x.reshape(shape)
#     dec_x = dec_x.permute(0, 4 1, 2, 3)
#     dec_x = dec_x.reshape(shape[0],shape[3],shape[1],shape[2])
    print("shape >>>>>>>>>>>>>> ",)
    dec_x = dec_x.reshape(shape[0],3,32,32)
    print('train imgs after reshape ',dec_x.shape)

    batch_size = 100
    num_workers = 0

    #torch.Tensor returns float so if want long then use torch.tensor
    tensor_x = torch.Tensor(dec_x)
    tensor_y = torch.tensor(dec_y,dtype=torch.long)
    mnist_bal = TensorDataset(tensor_x,tensor_y)
    train_loader = torch.utils.data.DataLoader(mnist_bal,
        batch_size=batch_size,shuffle=True,num_workers=num_workers)

    best_loss = np.inf

    t0 = time.time()
    if args['train']:
        enc_optim = torch.optim.Adam(encoder.parameters(), lr = args['lr'])
        dec_optim = torch.optim.Adam(decoder.parameters(), lr = args['lr'])

        for epoch in range(args['epochs']):
            train_loss = 0.0
            tmse_loss = 0.0
            tdiscr_loss = 0.0
            # train for one epoch -- set nets to train mode
            encoder.train()
            decoder.train()

            for images,labs in train_loader:

                # zero gradients for each batch
                encoder.zero_grad()
                decoder.zero_grad()
                #print(images)
                images, labs = images.to(device), labs.to(device)
                #print('images ',images.size())
                labsn = labs.detach().cpu().numpy()
                #print('labsn ',labsn.shape, labsn)

                # run images
                z_hat = encoder(images)

                x_hat = decoder(z_hat) #decoder outputs tanh
                #print('xhat ', x_hat.size())
                #print(x_hat)
                mse = criterion(x_hat,images)
                #print('mse ',mse)

                resx = []
                resy = []

                tc = np.random.choice(10,1)
                #tc = 9
                xbeg = dec_x[dec_y == tc]
                ybeg = dec_y[dec_y == tc]
                xlen = len(xbeg)
                nsamp = min(xlen, 100)
                ind = np.random.choice(list(range(len(xbeg))),nsamp,replace=False)
                xclass = xbeg[ind]
                yclass = ybeg[ind]

                xclen = len(xclass)
                #print('xclen ',xclen)
                xcminus = np.arange(1,xclen)
                #print('minus ',xcminus.shape,xcminus)

                xcplus = np.append(xcminus,0)
                #print('xcplus ',xcplus)
                xcnew = (xclass[[xcplus],:])
                #xcnew = np.squeeze(xcnew)
                xcnew = xcnew.reshape(xcnew.shape[1],xcnew.shape[2],xcnew.shape[3],xcnew.shape[4])
                #print('xcnew ',xcnew.shape)

                xcnew = torch.Tensor(xcnew)
                xcnew = xcnew.to(device)

                #encode xclass to feature space
                xclass = torch.Tensor(xclass)
                xclass = xclass.to(device)
                xclass = encoder(xclass)
                #print('xclass ',xclass.shape)

                xclass = xclass.detach().cpu().numpy()

                xc_enc = (xclass[[xcplus],:])
                xc_enc = np.squeeze(xc_enc)
                #print('xc enc ',xc_enc.shape)

                xc_enc = torch.Tensor(xc_enc)
                xc_enc = xc_enc.to(device)

                ximg = decoder(xc_enc)

                mse2 = criterion(ximg,xcnew)

                comb_loss = mse2 + mse
                comb_loss.backward()

                enc_optim.step()
                dec_optim.step()

                train_loss += comb_loss.item()*images.size(0)
                tmse_loss += mse.item()*images.size(0)
                tdiscr_loss += mse2.item()*images.size(0)

            # print avg training statistics
            train_loss = train_loss/len(train_loader)
            tmse_loss = tmse_loss/len(train_loader)
            tdiscr_loss = tdiscr_loss/len(train_loader)
            print('Epoch: {} \tTrain Loss: {:.6f} \tmse loss: {:.6f} \tmse2 loss: {:.6f}'.format(epoch,
                    train_loss,tmse_loss,tdiscr_loss))

            #store the best encoder and decoder models
            #here, /crs5 is a reference to 5 way cross validation, but is not
            #necessary for illustration purposes
            if train_loss < best_loss:
                print('Saving..')
#                 path_enc = "C:\\Users\\Research6\\Desktop\\smote"  + '\\bst_enc.pth'
#                 path_dec = "C:\\Users\\Research6\\Desktop\\smote"  + '\\bst_dec.pth'
                path_enc = path + '\\bst_enc.pth'
                path_dec = path + '\\bst_dec.pth'
                #    path_enc = '/content/gdrive/My Drive/smote/' \
                #     + str(i) + '/bst_enc.pth'
                # path_dec = '/content/gdrive/My Drive/smote/' \
                #     + str(i) + '/bst_dec.pth'

                torch.save(encoder.state_dict(), path_enc)
                torch.save(decoder.state_dict(), path_dec)

                best_loss = train_loss

        #in addition, store the final model (may not be the best) for
        #informational purposes
        path_enc = path + '\\f_enc.pth'
        path_dec = path  + '\\f_dec.pth'
        print(path_enc)
        print(path_dec)
        torch.save(encoder.state_dict(), path_enc)
        torch.save(decoder.state_dict(), path_dec)
        print()

    t1 = time.time()
    print('total time(min): {:.2f}'.format((t1 - t0)/60))

t4 = time.time()
print('final time(min): {:.2f}'.format((t4 - t3)/60))

Output:
I have the same issue on pytorch, can you try to uncomment the following line below inside nn.sequential() of decoder.
#3d and 32 by 32.
nn.Conv2d(self.dim_h * 4, self.dim_h * 8, 4, 1, 0, bias=False)