Instruction: why is ray Tune with pytorch HPO error 'trials did not complete, incomplete trials'?

Input:
Could someone explain why this code (that I took from here):
## Standard libraries
import os
import json
import math
import numpy as np
import time

## Imports for plotting
import matplotlib.pyplot as plt
#%matplotlib inline
#from IPython.display import set_matplotlib_formats
#set_matplotlib_formats('svg', 'pdf') # For export
from matplotlib.colors import to_rgb
import matplotlib
matplotlib.rcParams['lines.linewidth'] = 2.0
import seaborn as sns
sns.reset_orig()
sns.set()
import torch_geometric
import torch_geometric.nn as geom_nn
import torch_geometric.data as geom_data
from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint
## Progress bar
from tqdm.notebook import tqdm

## PyTorch
import torch
import torchmetrics
from torchmetrics.functional import precision_recall
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as data
import torch.optim as optim
# Torchvision
import torchvision
from torchvision.datasets import CIFAR10
from torchvision import transforms
# PyTorch Lightning
import pytorch_lightning as pl
from ray import tune

def __init__(self, config):
  super(LightningMNISTClassifier, self).__init__()
  self.layer_1_size = config["layer_1_size"]
  self.layer_2_size = config["layer_2_size"]
  self.lr = config["lr"]
  self.batch_size = config["batch_size"]

from ray.tune.integration.pytorch_lightning import TuneReportCallback
callback = TuneReportCallback(
    {
        "loss": "val_loss",
        "mean_accuracy": "val_accuracy"
    },
    on="validation_end")

def train_tune(config, epochs=10, gpus=0):
  model = LightningMNISTClassifier(config)
  trainer = pl.Trainer(
    max_epochs=epochs,
    gpus=gpus,
    progress_bar_refresh_rate=0,
    callbacks=[callback])
  trainer.fit(model)

config = {
  "layer_1_size": tune.choice([32, 64, 128]),
  "layer_2_size": tune.choice([64, 128, 256]),
  "lr": tune.loguniform(1e-4, 1e-1),
  "batch_size": tune.choice([32, 64, 128])
}

def train_tune(config, epochs=10, gpus=0):
  model = LightningMNISTClassifier(config)
  trainer = pl.Trainer(
    max_epochs=epochs,
    gpus=gpus,
    progress_bar_refresh_rate=0,
    callbacks=[callback])
  trainer.fit(model)

from functools import partial
tune.run(
  partial(train_tune, epochs=10, gpus=0),
  config=config,
  num_samples=10)

generates this error:
Traceback (most recent call last):
  File "example_hpo_working.py", line 89, in <module>
    num_samples=10)
  File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/tune.py", line 741, in run
    raise TuneError("Trials did not complete", incomplete_trials)
ray.tune.error.TuneError: ('Trials did not complete', [train_tune_6f362_00000, train_tune_6f362_00001, train_tune_6f362_00002, train_tune_6f362_00003, train_tune_6f362_00004, train_tune_6f362_00005, train_tune_6f362_00006, train_tune_6f362_00007, train_tune_6f362_00008, train_tune_6f362_00009])

I can see a similar question was asked here but not answered (the ultimate aim is to use ray hyperparameter optimisation with a pytorch network).
This is the full trace from the code:
2022-08-16 15:44:08,204 WARNING function_runner.py:604 -- Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
2022-08-16 15:44:08,411 ERROR syncer.py:147 -- Log sync requires rsync to be installed.
== Status ==
Memory usage on this node: 16.8/86.4 GiB
Using FIFO scheduling algorithm.
Resources requested: 1.0/64 CPUs, 0/0 GPUs, 0.0/62.79 GiB heap, 0.0/9.31 GiB objects
Result logdir: /root/ray_results/train_tune_2022-08-16_15-44-08
Number of trials: 10/10 (9 PENDING, 1 RUNNING)
+------------------------+----------+------------------+--------------+----------------+----------------+-------------+
| Trial name             | status   | loc              |   batch_size |   layer_1_size |   layer_2_size |          lr |
|------------------------+----------+------------------+--------------+----------------+----------------+-------------|
| train_tune_43fd5_00000 | RUNNING  | 172.17.0.2:41684 |           64 |             64 |            256 | 0.00233834  |
| train_tune_43fd5_00001 | PENDING  |                  |           64 |             64 |            256 | 0.00155955  |
| train_tune_43fd5_00002 | PENDING  |                  |          128 |            128 |             64 | 0.00399358  |
| train_tune_43fd5_00003 | PENDING  |                  |          128 |            128 |             64 | 0.000184477 |

...deleted a few similar lines here

..and then there's:

(func pid=41684) 2022-08-16 15:44:10,774        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41684) Traceback (most recent call last):
(func pid=41684)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41684)     self._entrypoint()
(func pid=41684)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41684)     self._status_reporter.get_checkpoint(),
(func pid=41684)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41684)     return method(self, *_args, **_kwargs)
(func pid=41684)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41684)     output = fn()
(func pid=41684)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41684)     model = LightningMNISTClassifier(config)
(func pid=41684) NameError: name 'LightningMNISTClassifier' is not defined
2022-08-16 15:44:10,977 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00000: Error processing event.
NoneType: None
Result for train_tune_43fd5_00000:
  date: 2022-08-16_15-44-10
  experiment_id: c8977e85cbf84a9badff15fb2de6f516
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41684
  timestamp: 1660664650
  trial_id: 43fd5_00000

(func pid=41722) 2022-08-16 15:44:13,241        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41722) Traceback (most recent call last):
(func pid=41722)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41722)     self._entrypoint()
(func pid=41722)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41722)     self._status_reporter.get_checkpoint(),
(func pid=41722)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41722)     return method(self, *_args, **_kwargs)
(func pid=41722)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41722)     output = fn()
(func pid=41722)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41722)     model = LightningMNISTClassifier(config)
(func pid=41722) NameError: name 'LightningMNISTClassifier' is not defined
(func pid=41720) 2022-08-16 15:44:13,253        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41720) Traceback (most recent call last):
(func pid=41720)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41720)     self._entrypoint()
(func pid=41720)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41720)     self._status_reporter.get_checkpoint(),
(func pid=41720)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41720)     return method(self, *_args, **_kwargs)
(func pid=41720)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41720)     output = fn()
(func pid=41720)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41720)     model = LightningMNISTClassifier(config)
(func pid=41720) NameError: name 'LightningMNISTClassifier' is not defined
(func pid=41718) 2022-08-16 15:44:13,253        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41718) Traceback (most recent call last):
(func pid=41718)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41718)     self._entrypoint()
(func pid=41718)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41718)     self._status_reporter.get_checkpoint(),
(func pid=41718)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41718)     return method(self, *_args, **_kwargs)
(func pid=41718)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41718)     output = fn()
(func pid=41718)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41718)     model = LightningMNISTClassifier(config)
(func pid=41718) NameError: name 'LightningMNISTClassifier' is not defined
(func pid=41734) 2022-08-16 15:44:13,340        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41734) Traceback (most recent call last):
(func pid=41734)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41734)     self._entrypoint()
(func pid=41734)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41734)     self._status_reporter.get_checkpoint(),
(func pid=41734)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41734)     return method(self, *_args, **_kwargs)
(func pid=41734)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41734)     output = fn()
(func pid=41734)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41734)     model = LightningMNISTClassifier(config)
(func pid=41734) NameError: name 'LightningMNISTClassifier' is not defined
(func pid=41732) 2022-08-16 15:44:13,325        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41732) Traceback (most recent call last):
(func pid=41732)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41732)     self._entrypoint()
(func pid=41732)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41732)     self._status_reporter.get_checkpoint(),
(func pid=41732)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41732)     return method(self, *_args, **_kwargs)
(func pid=41732)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41732)     output = fn()
(func pid=41732)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41732)     model = LightningMNISTClassifier(config)
(func pid=41732) NameError: name 'LightningMNISTClassifier' is not defined
(func pid=41728) 2022-08-16 15:44:13,309        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41728) Traceback (most recent call last):
(func pid=41728)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41728)     self._entrypoint()
(func pid=41728)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41728)     self._status_reporter.get_checkpoint(),
(func pid=41728)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41728)     return method(self, *_args, **_kwargs)
(func pid=41728)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41728)     output = fn()
(func pid=41728)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41728)     model = LightningMNISTClassifier(config)
(func pid=41728) NameError: name 'LightningMNISTClassifier' is not defined
(func pid=41730) 2022-08-16 15:44:13,272        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41730) Traceback (most recent call last):
(func pid=41730)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41730)     self._entrypoint()
(func pid=41730)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41730)     self._status_reporter.get_checkpoint(),
(func pid=41730)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41730)     return method(self, *_args, **_kwargs)
(func pid=41730)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41730)     output = fn()
(func pid=41730)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41730)     model = LightningMNISTClassifier(config)
(func pid=41730) NameError: name 'LightningMNISTClassifier' is not defined
2022-08-16 15:44:13,444 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00003: Error processing event.
NoneType: None
Result for train_tune_43fd5_00003:
  date: 2022-08-16_15-44-13
  experiment_id: 02204d81b72943e3bbfcc822d35f02a0
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41722
  timestamp: 1660664653
  trial_id: 43fd5_00003

(func pid=41724) 2022-08-16 15:44:13,457        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41724) Traceback (most recent call last):
(func pid=41724)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41724)     self._entrypoint()
(func pid=41724)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41724)     self._status_reporter.get_checkpoint(),
(func pid=41724)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41724)     return method(self, *_args, **_kwargs)
(func pid=41724)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41724)     output = fn()
(func pid=41724)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41724)     model = LightningMNISTClassifier(config)
(func pid=41724) NameError: name 'LightningMNISTClassifier' is not defined
== Status ==
Current time: 2022-08-16 15:44:13 (running for 00:00:05.24)
Memory usage on this node: 17.6/86.4 GiB
Using FIFO scheduling algorithm.
Resources requested: 8.0/64 CPUs, 0/0 GPUs, 0.0/62.79 GiB heap, 0.0/9.31 GiB objects
Result logdir: /root/ray_results/train_tune_2022-08-16_15-44-08
Number of trials: 10/10 (2 ERROR, 8 RUNNING)
+------------------------+----------+------------------+--------------+----------------+----------------+-------------+
| Trial name             | status   | loc              |   batch_size |   layer_1_size |   layer_2_size |          lr |
|------------------------+----------+------------------+--------------+----------------+----------------+-------------|
| train_tune_43fd5_00001 | RUNNING  | 172.17.0.2:41718 |           64 |             64 |            256 | 0.00155955  |
| train_tune_43fd5_00002 | RUNNING  | 172.17.0.2:41720 |          128 |            128 |             64 | 0.00399358  |
| train_tune_43fd5_00004 | RUNNING  | 172.17.0.2:41724 |          128 |             64 |            128 | 0.0221855   |
| train_tune_43fd5_00005 | RUNNING  | 172.17.0.2:41726 |           64 |            128 |            128 | 0.00041038  |
| train_tune_43fd5_00006 | RUNNING  | 172.17.0.2:41728 |           64 |             64 |            256 | 0.0105243   |
| train_tune_43fd5_00007 | RUNNING  | 172.17.0.2:41730 |          128 |             32 |            256 | 0.000929454 |
| train_tune_43fd5_00008 | RUNNING  | 172.17.0.2:41732 |           64 |             64 |            128 | 0.00176483  |
| train_tune_43fd5_00009 | RUNNING  | 172.17.0.2:41734 |          128 |             32 |            256 | 0.000113077 |
| train_tune_43fd5_00000 | ERROR    | 172.17.0.2:41684 |           64 |             64 |            256 | 0.00233834  |
| train_tune_43fd5_00003 | ERROR    | 172.17.0.2:41722 |          128 |            128 |             64 | 0.000184477 |
+------------------------+----------+------------------+--------------+----------------+----------------+-------------+
Number of errored trials: 2
+------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name             |   # failures | error file                                                                                |
|------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| train_tune_43fd5_00000 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00000_0_batch_size=64,layer_1_size=64,layer_2_size=256,lr=0.0023_2022-08-16_15-44-08/error.txt  |
| train_tune_43fd5_00003 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00003_3_batch_size=128,layer_1_size=128,layer_2_size=64,lr=0.0002_2022-08-16_15-44-10/error.txt |
+------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+

2022-08-16 15:44:13,487 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00001: Error processing event.
NoneType: None
Result for train_tune_43fd5_00001:
  date: 2022-08-16_15-44-13
  experiment_id: e738348e77c64919931d70c916cbfaf8
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41718
  timestamp: 1660664653
  trial_id: 43fd5_00001

2022-08-16 15:44:13,490 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00007: Error processing event.
NoneType: None
Result for train_tune_43fd5_00007:
  date: 2022-08-16_15-44-13
  experiment_id: f79be7b9e98a43f1a41893071c4e1f6b
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41730
  timestamp: 1660664653
  trial_id: 43fd5_00007

2022-08-16 15:44:13,493 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00002: Error processing event.
NoneType: None
Result for train_tune_43fd5_00002:
  date: 2022-08-16_15-44-13
  experiment_id: 8e7422287e3e44f9b2e7b249a8ae18cd
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41720
  timestamp: 1660664653
  trial_id: 43fd5_00002

2022-08-16 15:44:13,512 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00006: Error processing event.
NoneType: None
Result for train_tune_43fd5_00006:
  date: 2022-08-16_15-44-13
  experiment_id: 2d56b152a6a34e1f9e26dad1aec25d00
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41728
  timestamp: 1660664653
  trial_id: 43fd5_00006

2022-08-16 15:44:13,527 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00008: Error processing event.
NoneType: None
Result for train_tune_43fd5_00008:
  date: 2022-08-16_15-44-13
  experiment_id: b2158026b3b947bfbb9c3da4e6f7b977
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41732
  timestamp: 1660664653
  trial_id: 43fd5_00008

2022-08-16 15:44:13,543 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00009: Error processing event.
NoneType: None
Result for train_tune_43fd5_00009:
  date: 2022-08-16_15-44-13
  experiment_id: 6b5a73f09241440085bd6c09f6f681e9
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41734
  timestamp: 1660664653
  trial_id: 43fd5_00009

(func pid=41726) 2022-08-16 15:44:13,484        ERROR function_runner.py:286 -- Runner Thread raised error.
(func pid=41726) Traceback (most recent call last):
(func pid=41726)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 277, in run
(func pid=41726)     self._entrypoint()
(func pid=41726)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 352, in entrypoint
(func pid=41726)     self._status_reporter.get_checkpoint(),
(func pid=41726)   File "/root/miniconda3/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 462, in _resume_span
(func pid=41726)     return method(self, *_args, **_kwargs)
(func pid=41726)   File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 645, in _trainable_func
(func pid=41726)     output = fn()
(func pid=41726)   File "example_hpo_working.py", line 76, in train_tune
(func pid=41726)     model = LightningMNISTClassifier(config)
(func pid=41726) NameError: name 'LightningMNISTClassifier' is not defined
2022-08-16 15:44:13,660 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00004: Error processing event.
NoneType: None
Result for train_tune_43fd5_00004:
  date: 2022-08-16_15-44-13
  experiment_id: 60f51e072c7942bdb5d9298e0e147555
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41724
  timestamp: 1660664653
  trial_id: 43fd5_00004

2022-08-16 15:44:13,687 ERROR trial_runner.py:886 -- Trial train_tune_43fd5_00005: Error processing event.
NoneType: None
Result for train_tune_43fd5_00005:
  date: 2022-08-16_15-44-13
  experiment_id: 79701d1c19ac4c55b5a73746c1872724
  hostname: 0e26c6a24ffa
  node_ip: 172.17.0.2
  pid: 41726
  timestamp: 1660664653
  trial_id: 43fd5_00005

== Status ==
Current time: 2022-08-16 15:44:13 (running for 00:00:05.46)
Memory usage on this node: 16.4/86.4 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0/0 GPUs, 0.0/62.79 GiB heap, 0.0/9.31 GiB objects
Result logdir: /root/ray_results/train_tune_2022-08-16_15-44-08
Number of trials: 10/10 (10 ERROR)
+------------------------+----------+------------------+--------------+----------------+----------------+-------------+
| Trial name             | status   | loc              |   batch_size |   layer_1_size |   layer_2_size |          lr |
|------------------------+----------+------------------+--------------+----------------+----------------+-------------|
| train_tune_43fd5_00000 | ERROR    | 172.17.0.2:41684 |           64 |             64 |            256 | 0.00233834  |
| train_tune_43fd5_00001 | ERROR    | 172.17.0.2:41718 |           64 |             64 |            256 | 0.00155955  |
| train_tune_43fd5_00002 | ERROR    | 172.17.0.2:41720 |          128 |            128 |             64 | 0.00399358  |
| train_tune_43fd5_00003 | ERROR    | 172.17.0.2:41722 |          128 |            128 |             64 | 0.000184477 |
| train_tune_43fd5_00004 | ERROR    | 172.17.0.2:41724 |          128 |             64 |            128 | 0.0221855   |
| train_tune_43fd5_00005 | ERROR    | 172.17.0.2:41726 |           64 |            128 |            128 | 0.00041038  |
| train_tune_43fd5_00006 | ERROR    | 172.17.0.2:41728 |           64 |             64 |            256 | 0.0105243   |
| train_tune_43fd5_00007 | ERROR    | 172.17.0.2:41730 |          128 |             32 |            256 | 0.000929454 |
| train_tune_43fd5_00008 | ERROR    | 172.17.0.2:41732 |           64 |             64 |            128 | 0.00176483  |
| train_tune_43fd5_00009 | ERROR    | 172.17.0.2:41734 |          128 |             32 |            256 | 0.000113077 |
+------------------------+----------+------------------+--------------+----------------+----------------+-------------+
Number of errored trials: 10
+------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name             |   # failures | error file                                                                                |
|------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| train_tune_43fd5_00000 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00000_0_batch_size=64,layer_1_size=64,layer_2_size=256,lr=0.0023_2022-08-16_15-44-08/error.txt  |
| train_tune_43fd5_00001 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00001_1_batch_size=64,layer_1_size=64,layer_2_size=256,lr=0.0016_2022-08-16_15-44-10/error.txt  |
| train_tune_43fd5_00002 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00002_2_batch_size=128,layer_1_size=128,layer_2_size=64,lr=0.0040_2022-08-16_15-44-10/error.txt |
| train_tune_43fd5_00003 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00003_3_batch_size=128,layer_1_size=128,layer_2_size=64,lr=0.0002_2022-08-16_15-44-10/error.txt |
| train_tune_43fd5_00004 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00004_4_batch_size=128,layer_1_size=64,layer_2_size=128,lr=0.0222_2022-08-16_15-44-10/error.txt |
| train_tune_43fd5_00005 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00005_5_batch_size=64,layer_1_size=128,layer_2_size=128,lr=0.0004_2022-08-16_15-44-10/error.txt |
| train_tune_43fd5_00006 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00006_6_batch_size=64,layer_1_size=64,layer_2_size=256,lr=0.0105_2022-08-16_15-44-10/error.txt  |
| train_tune_43fd5_00007 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00007_7_batch_size=128,layer_1_size=32,layer_2_size=256,lr=0.0009_2022-08-16_15-44-10/error.txt |
| train_tune_43fd5_00008 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00008_8_batch_size=64,layer_1_size=64,layer_2_size=128,lr=0.0018_2022-08-16_15-44-10/error.txt  |
| train_tune_43fd5_00009 |            1 | /root/ray_results/train_tune_2022-08-16_15-44-08/train_tune_43fd5_00009_9_batch_size=128,layer_1_size=32,layer_2_size=256,lr=0.0001_2022-08-16_15-44-10/error.txt |
+------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Traceback (most recent call last):
  File "example_hpo_working.py", line 89, in <module>
    num_samples=10)
  File "/root/miniconda3/lib/python3.7/site-packages/ray/tune/tune.py", line 741, in run
    raise TuneError("Trials did not complete", incomplete_trials)
ray.tune.error.TuneError: ('Trials did not complete', [train_tune_43fd5_00000, train_tune_43fd5_00001, train_tune_43fd5_00002, train_tune_43fd5_00003, train_tune_43fd5_00004, train_tune_43fd5_00005, train_tune_43fd5_00006, train_tune_43fd5_00007, train_tune_43fd5_00008, train_tune_43fd5_00009])

Output:
Is there a longer stacktrace where the real error is printed?
Also could you go to the result folder and see the error file?
Usually result folder is under ~/ray_results.


Instruction: How to convert image from folder into tensors using torch?

Input:
I'm trying to convert images in a folder to tensors, save it and load them later, as shown below
transform = transforms.Compose([
  transforms.ToTensor()])
dataset = datasets.ImageFolder(
  r'imagedata', transform=transform)
torch.save(dataset, 'train_data.pt')

But I get a value error when trying to load the trained file as below:
train_codes = torch.Tensor(torch.load(os.path.join(self.data_dir, "train_data.pt")))

ValueError: only one element tensors can be converted to Python scalars
Any help or suggestion to fix this will be highly appreciated.

Output:
You met this problem because train_data.pt was not saved as a Tensor, since that variable was read the data by ImageFolder which was inherited from DatasetFolder, it should be loaded and used as a Torch Dataset. The example below use DataLoader as documents:
import torch
from torchvision import transforms, datasets

# Saving part
transform = transforms.Compose([
                                transforms.ToTensor()
                               ])
dataset = datasets.ImageFolder(r'imagedata', transform=transform)
torch.save(dataset,'train_data.pt')

# Loading part
data = torch.load(torch.load(os.path.join(self.data_dir, "train_data.pt")))
loader = torch.utils.data.DataLoader(data, batch_size = 32)
for image, label in loader:
   # Processing....


Instruction: Pytorch model function gives Error :'NoneType' object has no attribute 'size'

Input:
I trying to run below code in my image segmentation problem without CUDA as I don't have GPU. I have trained my model on CPU using pytorch but on prediction level I'm getting
AttributeError: 'NoneType' object has no attribute 'size'

Here's the code:
idx = 20
model.load_state_dict(torch.load('/content/best_model.pt'))

image, mask = validset[idx]
image = image.unsqueeze_(0)
print(type(image))
# logits_mask = model(image.to(DEVICE).unsqueeze(0)) # (c,h,w) -> (1,c,h,w)
logits_mask = model(image) # (c,h,w) py-> (1,c,h,w)

The resulting error, from the output, is at line number 8:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-56-edf3f0fae49c> in <module>
      6 print(type(image))
      7 # logits_mask = model(image.to(DEVICE).unsqueeze(0)) # (c,h,w) -> (1,c,h,w)
----> 8 logits_mask = model(image) # (c,h,w) py-> (1,c,h,w)
      9
     10 pred_mask = torch.sigmoid(logits_mask)

3 frames
/usr/local/lib/python3.7/dist-packages/segmentation_models_pytorch/losses/dice.py in forward(self, y_pred, y_true)
     57     def forward(self, y_pred: torch.Tensor, y_true: torch.Tensor) -> torch.Tensor:
     58
---> 59         assert y_true.size(0) == y_pred.size(0)
     60
     61         if self.from_logits:

AttributeError: 'NoneType' object has no attribute 'size'

Output:
assert y_true.size(0) == y_pred.size(0) erroring signifies that either y_true or y_pred are None, so you can try checking the types of image, model(image), & mask respectively.
IMO, this might be the root cause: image = image.unsqueeze_(0)
unsqueze_ is an inplace operator, which means it will return nothing and change the tensor image inplace.


Instruction: BertModel weights are randomly initialized?

Input:
recently, I've been trying to re-implement DiffCSE
During refactoring the codes that the authors uploaded on Github,  I've run into some issues.
I have 2 questions
1.
If I set seed like set_seed(30), I was under the impression that  the model has the same initialized weights, thus making the same result when training.  But It feels like I was wrong
for example,
config = AutoConfig.from_pretrained('bert-base-uncased')
a = BertModel(config)
b = BertModel(config)
a_query =a.encoder.layer[0].attention.self.query.weight
b_query =b.encoder.layer[0].attention.self.query.weight
a_query == b_query
# tensor([[False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        ...,
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False]])

print(a_query, b_query)
Parameter containing:
tensor([[ 0.0168, -0.0072,  0.0141,  ...,  0.0060, -0.0098, -0.0361],
        [ 0.0121, -0.0106,  0.0169,  ..., -0.0512,  0.0154, -0.0251],
        [ 0.0252,  0.0375,  0.0215,  ..., -0.0097, -0.0009, -0.0102],
        ...,
        [ 0.0038,  0.0120, -0.0205,  ..., -0.0082, -0.0066,  0.0125],
        [ 0.0032, -0.0330,  0.0073,  ...,  0.0072,  0.0484,  0.0143],
        [-0.0153,  0.0207, -0.0086,  ..., -0.0087, -0.0032,  0.0022]],
       requires_grad=True) Parameter containing:
tensor([[ 0.0239,  0.0236,  0.0181,  ..., -0.0331,  0.0062,  0.0142],
        [-0.0116,  0.0417, -0.0379,  ...,  0.0059,  0.0207,  0.0155],
        [ 0.0178,  0.0017,  0.0064,  ..., -0.0007,  0.0405, -0.0170],
        ...,
        [ 0.0115,  0.0039, -0.0508,  ...,  0.0187,  0.0043, -0.0048],
        [ 0.0025, -0.0079, -0.0132,  ..., -0.0003, -0.0079,  0.0320],
        [-0.0105, -0.0097, -0.0076,  ...,  0.0214, -0.0068,  0.0016]],
       requires_grad=True)

I can't understand why it happens. Also, Every time I execute this code, the weights are different from each case.
2.
There are many models provided by Huggingface. When it comes to BERT, they have BertModel, BertForPretraining, BertForMaskedLM,, etc. As far as I know, the only difference between each Bert model is whether they have heads on the top layer or not.
Then, the heads are also pretrained?? or just randomly initailzed weights and provieded for users' convenience.??

Output:
You have a small misunderstanding of how seeds work. The seed defines how the random values are sampled, it doesn't reset after each sample. This means that the sequences sampled will be the same when starting from the seed. For example, if you have a code like:
seed = 1
sample = sample_4_values()

You should always get the same four values because the seed defined this sequence.
In your case you define 2 BERT models without resetting the seed to the starting point for each sample isn't the same!
In order to get the same weights to reset the seed before each initialization of BERT
##Edit
To better understand what the seed does you need to think about it as a starting point. Imagine that setting the seed to 30 tells the computer to sample the following numbers: 1,2,3,5,6
Calling the sample function 1 time will return 1. Calling it again will return 2 and so on. What you are basically doing is sampling 2 times but each time your starting point is different.


Instruction: pytorch tensor sort rows based on column

Input:
In a 2D tensor like so
tensor([[0.8771, 0.0976, 0.8186],
        [0.7044, 0.4783, 0.0350],
        [0.4239, 0.8341, 0.3693],
        [0.5568, 0.9175, 0.0763],
        [0.0876, 0.1651, 0.2776]])

How do you sort the rows based off the values in a column? For instance if we were to sort based off the last column, I would expect the rows to be such...
tensor([[0.7044, 0.4783, 0.0350],
        [0.5568, 0.9175, 0.0763],
        [0.0876, 0.1651, 0.2776],
        [0.4239, 0.8341, 0.3693],
        [0.8771, 0.0976, 0.8186]])

Values in the last column are now in ascending order.

Output:
a = <your tensor>
ind = a[:,-1].argsort(dim=0)
a[ind]

argsort "Returns the indices that sort a tensor along a given dimension in ascending order by value." So, basically, you get sorting indices for the last column and reorder the rows according to these indices.


Instruction: PyTorch Lightning - How to automatically reload last checkpoint when loss unexpectedly spikes?

Input:
I'm facing a problem where during training, my loss will unexpectedly spike, like so:

When this happens, I want to automatically reload the last checkpoint, reset the optimizer and resume training. How do I do this?
Edit: I tried training with fp64 precision and the unstable learning problem still occurred albeit later in training.

Output: you could write a callback where you can check the spike and load the last checkpoint. Please, let me know if this helps!


Instruction: ImportError about Detectron2

Input:
I was trying to do semantic segmentation using Detectron2, but some tricky errors occurred when I ran my program. It seems there might be some problems in my environment.
Does anyone know how to fix it?

ImportError: cannot import name 'is_fx_tracing' from 'torch.fx._symbolic_trace' (/home/eric/anaconda3/envs/detectron_env/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py)

Output:
This seems to be an issue with the latest commit of detectron2, you can use the previous commit of detectron2 while installing to avoid this error.
pip install 'git+https://github.com/facebookresearch/detectron2.git@5aeb252b194b93dc2879b4ac34bc51a31b5aee13'

The issue is resolved in the latest commit of detectron2


Instruction: ValueError: The following `model_kwargs` are not used by the model: ['encoder_outputs'] (note: typos in the generate arguments will also show up

Input:
When I try to run my  code for Donut for DocVQA model, I got the following error
"""Test"""
from donut import DonutModel
from PIL import Image
import torch

model = DonutModel.from_pretrained(
    "naver-clova-ix/donut-base-finetuned-cord-v2")
if torch.cuda.is_available():
    model.half()
    device = torch.device("cuda")
    model.to(device)
else:
    model.encoder.to(torch.bfloat16)
model.eval()
image = Image.open(
    "./src/png-report-page_capone_v1_August_2017_GMS_Balance_Sheet_0.png").convert("RGB")
output = model.inference(image=image, prompt="<s_cord-v2>")

print(output)

The error`
ValueError: The following `model_kwargs` are not used by the model: ['encoder_outputs'] (note: typos in the generate arguments will also show up in this list)

Output:
Check the transformers library version.
For me, I reinstalled the 4.21.3 version and it worked.


Instruction: PyTorch Tensor methods to nn.Modules?

Input:
I'm programming some callable custom modules in PyTorch and I wanted to know if I'm doing it correctly. Here's an example scenario where I want to construct a module that takes a torch.Tensor as input, performs a learnable linear operation and outputs a diagonal covariance matrix to use in a multivariate distribution downstream.
class Exp(nn.Module):
    def forward(self, x):
        return x.exp()

class Diag(nn.Module):
    def forward(self, x):
        return x.diag_embed()

def init_model(input_size, output_size):
    log_variance_module = nn.Linear(input_size, output_size)
    diag_covariance_module = nn.Sequential(logvar_module, Exp(), Diag())
    return diag_covariance_module

model = init_model(5, 5)
cov = model(some_input_tensor)
dist = MultivariateNormal(some_mean, cov)

I know that this works, but is it the right design pattern? How is one recommended to approach these modules?

Output:
This looks like the correct design pattern.
Ideally, you would also write your main network as an nn.Module:
class Model(nn.Sequential):
    def __init__(self, input_size, output_size):
        logvar_module = nn.Linear(input_size, output_size)
        super().__init__(logvar_module, Exp(), Diag())


Instruction: Local maximums of sub-tensors by index tensor

Input:
I have a tensor x of shape (1,n), and another index tensor d of shape (1,k). I’m trying to find the maximums of k sub-tensors
x[0:d[0]], x[d[0]:d[1]], x[d[1]:d[2]], ..., x[d[-2]: d[-1]]

So the output is a tensor of shape (1,k) with k local maximums. I can implement a for loop, but that’s too slow. Can I do it in parallel in PyTorch (or Numpy)?

Output:
I found the answer thanks to user7138814. There is a SegmentCSR function in torch_scatter that does the job:
from torch_scatter import segment_csr

src = torch.randn(10, 6, 64)
indptr = torch.tensor([0, 2, 5, 6])
indptr = indptr.view(1, -1)  # Broadcasting in the first and last dim.

out = segment_csr(src, indptr, reduce="sum")

print(out.size())
torch.Size([10, 3, 64])

 output: torch.Size([10, 3, 64])


Instruction: How pytorch loss connect to model parameters?

Input:
I know that in PyTorch optimizer is connected to the model's parameters by
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

and inside the training loop we have to do the backward and update the gradient by execute this two lines
loss.backward()
optimizer.step()

But how does the loss actually connect to the model parameters? because we only define connection between optimizer and model and never define the connection between loss and the model.
And when we execute loss.backward(), how does PyTorch know that we will do backpropagation for our model?
I put the full code here for the context
import torch
import torch.nn as nn

X = torch.tensor([[1], [2], [3], [4]], dtype=torch.float32)
Y = torch.tensor([[2], [4], [6], [8]], dtype=torch.float32)
X_test = torch.tensor([[5]], dtype=torch.float32)

n_sample, n_feature = X.shape
input_size = n_feature
output_size = n_feature

model = nn.Linear(input_size, output_size)

# Training
learning_rate = 0.01
n_iters = 100

loss = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

# print(model(X_test))
print(f"Prediction before training f(5) = {model(X_test).item():.3f}")

for epoch in range(n_iters):
  y_pred = model(X)

  # compute loss
  l = loss(Y, y_pred)

  # gradient
  l.backward()

  # update gradient
  optimizer.step()

  # zero gradient
  optimizer.zero_grad()

  if epoch % 10 == 0:
    w, b = model.parameters()
    # print(model.parameters())
    print(f"Epoch {epoch + 1}, w = {w[0][0].item():.3f}, loss = {l:.5f}")

print(f"Prediction after training f(5) = {model(X_test).item():.3f}")

Output:
Q: When we execute loss.backward(), how does PyTorch know that we will do backpropagation for our model?
In the line l = loss(Y, y_pred), the predictions are used to calculate the loss. This effectively connects the model parameters with the loss such that loss.backward() can do the backpropagation for the network to compute the parameter gradients. Note that the tensors in model() have requires_grad=True, while this is not the case for the labels which do not need gradients. Through l.backward(), each tensor value that went into the loss calculation and requires a gradient (in our case that's the model parameters) is assigned a gradient. See the documentation for the grad attribute.
Q: But how does the loss actually connect to the model parameters?
The statement optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate) connects optimizer and model parameters.
Since the gradients computed through loss.backward() become attributes of the model parameters, they are accessible to the optimizer.


Instruction: Instance Norm: ValueError: Expected more than 1 spatial element when training, got input size torch.Size([128, 512, 1, 1])

Input:
I have a ResNet-18 working well. Now, I want to use InstanceNorm as normalization layer instead of BatchNorm, so I changed all the batchnorm layers in this way:
resnet18.bn1 = nn.InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)

resnet18.layer1[0].bn1 = nn.InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer1[0].bn2 = nn.InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer1[1].bn1 = nn.InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer1[1].bn2 = nn.InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)

resnet18.layer2[0].bn1 = nn.InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer2[0].bn2 = nn.InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer2[1].bn1 = nn.InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer2[1].bn2 = nn.InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer2[0].downsample[1] = nn.InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)

resnet18.layer3[0].bn1 = nn.InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer3[0].bn2 = nn.InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer3[1].bn1 = nn.InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer3[1].bn2 = nn.InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer3[0].downsample[1] = nn.InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)

resnet18.layer4[0].bn1 = nn.InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer4[0].bn2 = nn.InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer4[1].bn1 = nn.InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer4[1].bn2 = nn.InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
resnet18.layer4[0].downsample[1] = nn.InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)

resnet18.fc = nn.Linear(in_features=512, out_features=10, bias=True)

All the num_features are equal to the BatchNorm2d ones, I just changed BatchNorm2d into InstanceNorm2d. So my ResNet-18 is this:
ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): InstanceNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): InstanceNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): InstanceNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): InstanceNorm2d(512, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=512, out_features=10, bias=True)
)

I have the error in title. Do you know how can I fix?

Output: I was using CIFAR-10 with size 32x32. If I resize the images to 64x64 It works. This because resnet-18 reduces the filters to 1x1 and as the title says, InstanceNorm wants dimensions (H and W) > 1


Instruction: Lightning Flash error with SemanticSegmentationData (NameError: name 'K' is not defined)

Input:
I'm trying to perform an image segmentation task with Colab and Lightning Flash.
I'm installing Flash with:
!pip install lightning-flash

I'm trying to instanciate a Lightning Flash SemanticSegmentationData using from_folders method like this:
datamodule = SemanticSegmentationData.from_folders(
    train_folder=x_train_dir,
    train_target_folder=y_train_dir,
    val_folder=x_valid_dir,
    val_target_folder=y_valid_dir,
    test_folder=x_test_dir,
    test_target_folder=y_test_dir,
    transform_kwargs=dict(image_size=(256, 256)),
    num_classes=1,
    batch_size=16,
)

But I'm getting this error:
/usr/local/lib/python3.7/dist-packages/flash/image/segmentation/input_transform.py in train_per_sample_transform(self)
 49             [DataKeys.INPUT, DataKeys.TARGET],
 50             KorniaParallelTransforms(
---> 51                 K.geometry.Resize(self.image_size, interpolation="nearest"), K.augmentation.RandomHorizontalFlip(p=0.5)
 52             ),
 53         )
NameError: name 'K' is not defined

How can I solve this problem?

Output:
Initially, I was trying to solve the problem this way:
!pip install kornia
import kornia as K

This didn't solve the problem. Then, I opened an issue in their GitHub Issue #1423. With a help from https://github.com/krshrimali, I discovered that just installing kornia would solve the problem.
So, the solution is just:
!pip install kornia


Instruction: Is it possible to perform step according to batch size in pytorch?

Input:
I am iterating over training samples in batches, however last batch always returns fewer samples.
Is it possible to specify step size in torch according to the current batch length?
For example most batch are of size 64, last batch only 6 samples.
If I do the usual routine:
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

It seems that the last 6 samples carry the same weight when updating the gradients as the 64 sized batches, but in fact they should only carry about 1/10 weight due to fewer samples.
In Mxnet I could specify the step size accordingly but I don't know how to do it in torch.

Output:
You can define a custom loss function and then e.g. reweight it based on batch size
def reweighted_cross_entropy(my_outputs, my_labels):
    # compute batch size
    my_batch_size = my_outputs.size()[0]

    original_loss = nn.CrossEntropyLoss()
    loss = original_loss (my_outputs, my_labels)

    # reweight accordingly
    return my_batch_size * loss

if you are using something like gradient descent then it is easy to see that
[1/10 * lr] grad [loss] = lr * grad [ 1/10 loss]

so reweighting the loss will be equivalent to reweighting your learning rate. This won't be exactly true for more comlpex optimisers though but can be good enough in practise.


Instruction: How to apply random forests to the output produced by Bert?

Input:
I'm trying to get the output embeddings of a RoBERTa model, so I can train a random forests classifier on it for text classification (sentiment analysis). The original dataset this is based on is 500 news articles that each have a left/center/right bias rating. 80% of this dataset is training data, the other 20% is test data.
I run the following code for my training set:
# Tokenize sentences van trainingset
encoded_input = tokenizer(X_train, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling
sentence_embeddings = mean_pooling(model_output,encoded_input['attention_mask'])

## Normalize embeddings
sentence_embeddings = F.normalize(sentence_embeddings, p=3, dim=1)

start = True
numpy_emb = []

if not start:
    np_emb = sentence_embeddings.cpu().detach().numpy()
    numpy_emb = np.vstack([numpy_emb, np_emb])
else:
    start = False
    numpy_emb = np_emb = sentence_embeddings.cpu().detach().numpy()

Which gives me numpy_emb. Which I think are the embeddings that the RoBERTa model outputs.
When I print it, it gives me:
tensor([[ 0.5329, -0.1224],
        [ 0.5409, -0.0730],
        [ 0.4594, -0.1282],
        [ 0.5116, -0.0769],
        [ 0.4861, -0.0212],
        [ 0.5246, -0.0560],
        [ 0.5555, -0.0962],
        [ 0.4779, -0.0551],
        [ 0.5428, -0.0904],
        [ 0.5939, -0.0504],
        [ 0.5219, -0.1342],
        [ 0.4672, -0.0936],
        [ 0.5051, -0.0518],
        [ 0.5536, -0.1016],
        [ 0.4761, -0.0736],
        [ 0.4754, -0.0991],
        [ 0.5613, -0.0541],
        [ 0.5155,  0.0303],
        [ 0.6053,  0.0214],
        [ 0.4766, -0.1019],
        [ 0.4262, -0.0869],
        [ 0.3871, -0.0756],
        [ 0.5048, -0.0067],
        [ 0.5425, -0.1303],
        [ 0.5020, -0.0715],
    ...
        [ 0.5462, -0.0686],
        [ 0.5476, -0.1465],
        [ 0.4968, -0.0354],
        [ 0.5586, -0.1234],
        [ 0.5725, -0.0685]])

I then repeat this process for my test set as well, giving me another set of embeddings.
Then I try to train a random forests classifier using the embeddings given by the training set. But when I try to predict using the embeddings from my test set, I get very random results. Accuracy goes as low as 24% and as high as 58%. Is this because of the small amount of data that I have? Or is there something else I'm doing wrong?
I also have the suspicion that I can't properly link the output embedding to their respective label. Which would also explain the random results I get.
Code for random forests that I used:
from sklearn.ensemble import RandomForestClassifier
text_classifier = RandomForestClassifier(n_estimators=100, random_state=0)
text_classifier.fit(numpy_emb, y_train)

predictions = text_classifier.predict(numpy_emb_test)

#confusion matrix
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
print(confusion_matrix(y_test,predictions))
print(classification_report(y_test,predictions))

Output:
This shape doesn't look like proper embedding. For classification purpose, a usual approach to encoder-only models is just supplying the last hidden state as embeddings for the classifier, for example:
features = model_output[0][:,0,:].numpy()
text_classifier.fit(features, y_train)


Instruction: How to add a channel layer for a 3D image for 3D CNN

Input:
I am working on a medical dataset of 3d images the shapeimage.shape gives me (512,512,241) I assume it is height , width, depth but when I run 3D cnn i get this error Given groups=1, weight of size [32, 3, 3, 3, 3], expected input[1, 1, 241, 512, 512] to have 3 channels, but got 1 channels instead. what should be done? I am using pytorch.
Thank you in advance.

Output:
Your network expects each slice of the 3d volume to have three channels (RGB). You can simply convert grayscale (single channel) data to "fake" RGB by duplicating the single channel you have:
x_gray = ...  # your tensor of shape batch-1-241-512-512
x_fake_rgb = x_gray.expand(-1, 3, -1, -1, -1)

See expand for more details.


Instruction: AttributeError: module 'dill' has no attribute 'extend'

Input:
I have just installed Pytorch, using:
(base) C:\>pip3 install torch torchvision torchaudio
Requirement already satisfied: torch in c:\users\Emil\appdata\roaming\python\python38\site-packages (1.9.0)
Requirement already satisfied: torchvision in c:\users\Emil\appdata\roaming\python\python38\site-packages (0.10.0)
Requirement already satisfied: torchaudio in c:\users\Emil\anaconda3\lib\site-packages (0.12.1)
Requirement already satisfied: typing-extensions in c:\users\Emil\anaconda3\lib\site-packages (from torch) (3.7.4.3)
Requirement already satisfied: numpy in c:\users\Emil\anaconda3\lib\site-packages (from torchvision) (1.22.2)
Requirement already satisfied: pillow>=5.3.0 in c:\users\Emil\anaconda3\lib\site-packages (from torchvision) (8.0.1)

Then, I tried to import torch in Spyder but received the following error:
import torch
C:\Users\Emil\AppData\Roaming\Python\Python38\site-packages\torch\package\_mock_zipreader.py:17: UserWarning: Failed to initialize NumPy: numpy.core.multiarray failed to import (Triggered internally at  ..\torch\csrc\utils\tensor_numpy.cpp:67.)
  _dtype_to_storage = {data_type(0).dtype: data_type for data_type in _storages}
Traceback (most recent call last):

  File "<ipython-input-1-eb42ca6e4af3>", line 1, in <module>
    import torch

  File "C:\Users\Emil\AppData\Roaming\Python\Python38\site-packages\torch\__init__.py", line 705, in <module>
    import torch.utils.data

  File "C:\Users\Emil\AppData\Roaming\Python\Python38\site-packages\torch\utils\data\__init__.py", line 28, in <module>
    from torch.utils.data import datapipes

  File "C:\Users\Emil\AppData\Roaming\Python\Python38\site-packages\torch\utils\data\datapipes\__init__.py", line 1, in <module>
    from . import iter

  File "C:\Users\Emil\AppData\Roaming\Python\Python38\site-packages\torch\utils\data\datapipes\iter\__init__.py", line 8, in <module>
    from torch.utils.data.datapipes.iter.callable import \

  File "C:\Users\Emil\AppData\Roaming\Python\Python38\site-packages\torch\utils\data\datapipes\iter\callable.py", line 13, in <module>
    dill.extend(use_dill=False)

AttributeError: module 'dill' has no attribute 'extend'

What can I do to overcome this error?

Output:
Based on Mike McKerns' comment, I successfully used:
pip install dill --upgrade


Instruction: How to flush GPU memory using CUDA on WSL2

Input:
I have interrupted the training of the model in PyTorch on CUDA, which I've run on Windows Subsystem for Linux 2 (WSL2). The dedicated GPU memory of NVIDIA GeForce RTX 3080Ti was not flushed.

What I have tried:

gc.collect() and torch.cuda.empty_cache() does not resolve the problem (reference)

When running numba.cuda.select_device(0) to potentially cuda.close(), the notebook hangs (reference)

After running nvidia-smi to potentially reset the GPU (reference), the command prompt hangs

 Win  +  Ctrl  +  Shift  +  B  to reset the graphics stack in Windows does not help (reference)

Restarting the notebook kernel as well as restarting the notebook server does not help

Physical reset is not available

UPDATE:
Running nvidia-smi in the command prompt on Windows (not on WSL2) yields the following

Output:
I don't know your actual environment.
suppose that you use anaconda window-venv

On cmd >nvidia-smi shows following.

Check pid of  python process name(>envs\psychopy\python.exe).
On cmd taskkill /f /PID xxxx

this could be help.
and you don't want doing like this.
if you feeling annoying you can run the script on prompt, it would be automatically flushing gpu memory.


Instruction: When using torch.autocast, how do I force individual layers to float32

Input:
I'm trying to train a model in mixed precision. However, I want a few of the layers to be in full precision for stability reasons. How do I force an individual layer to be float32 when using torch.autocast? In particular, I'd like for this to be onnx compileable.
Is it something like:
with torch.cuda.amp.autocast(enabled=False, dtype=torch.float32):
    out = my_unstable_layer(inputs.float())

Edit:
Looks like this is indeed the official method. See the torch docs.

Output:
I think the motivation of torch.autocast is to automate the reduction of precision (not the increase).
If you have functions that need a particular dtype, you should consider using, custom_fwd
import torch
@torch.cuda.amp.custom_fwd(cast_inputs=torch.complex128)
def get_custom(x):
    print('  Decorated function received', x.dtype)
def regular_func(x):
    print('  Regular function received', x.dtype)
    get_custom(x)

x = torch.tensor(0.0, dtype=torch.half, device='cuda')
with torch.cuda.amp.autocast(False):
    print('autocast disabled')
    regular_func(x)
with torch.cuda.amp.autocast(True):
    print('autocast enabled')
    regular_func(x)

autocast disabled
  Regular function received torch.float16
  Decorated function received torch.float16
autocast enabled
  Regular function received torch.float16
  Decorated function received torch.complex128

Edit: Using torchscript
I am not sure how much you can rely on this, due to a comment in the documentation. However the comment is apparently outdated.
Here is an example where I trace the model with autocast enabled, feeze it and then I use it and the value is indeed cast to the specified type
class Cast(torch.nn.Module):
    @torch.cuda.amp.custom_fwd(cast_inputs=torch.float64)
    def forward(self, x):
        return x

with torch.cuda.amp.autocast(True):
    model = torch.jit.trace(Cast().eval(), x)
model = torch.jit.freeze(model)

x = torch.tensor(0.0, dtype=torch.half, device='cuda')
print(model(x).dtype)

torch.float64

But I suggest you to validate this approach before using it for a serious application.


Instruction: Cost of back-propagation for subset of DNN parameters

Input:
I am using pytorch for evaluating gradients of feed-forward network, but only for a subset of parameters, related to the first two layers.
Since backpropagation is carried backwards layer by layer, I wonder: why is it computationally faster than evaluating gradients of whole network?

Output:
Pytorch builds a computation graph for backward propagation that only contains the minimum nodes and edges to get the accumulated gradient for leaves that require gradient. Even if the first two layers require gradient, there are many tensors (intermediate tensors or frozen parameters tensors) that are unused and that are cut in the backward graph. Plus the built-in function AccumulatedGradient that stores the gradients in .grad attribute is call less time reducing the total computation time too.
Here you can see an example for an "AddBackward Node" where for instance A is an intermediate tensor computed with the first two layers and B is the 3rd (constant) layer that can be ignored.

An other example: if you have a matrix-matrix product (MmBackward Node) that uses an intermediate tensor that not depends on the 2 first layers. In this case the tensor itself is required to compute the backprop but the "previous" tensors that were used to compute it can be ignored in the graph.
To visualize the sub-graph that is actually computed (and compare when the model is unfrozen), you can use torchviz.


Instruction: PyTorch: How to create a Parameter without specifying the dimension

Input:
Say I want to defined a module. In this module, the __init__() function will create a Parameter called self.weight without known the input_dim of the module. My question is, how can I expand the self.weight and initialize it when I first call the forward() function?
For example, I want my module looks like this:
class MyModel(torch.nn.Module):
    def __init__(self, out_dim):
        super(MyModel, self).__init__()
        # I don't know the input_dim yet
        self.weight = torch.nn.Parameter(torch.FloatTensor(None, out_dim))
        self.init_weight = False

    def init_parameters(self, in_dim):
        # what should I do in this function?
        # Is this correct?
        self.weight = self.weight.expand(in_dim, -1)
        torch.nn.init.xvaier_normal_(self.weight)
        self.init_weight = True

    def forward(self, X):
        if not self.init_weight:
            # first call, so now I can initialize the weight since I know the input_dim
            self.init_parameters(X.shape[1])

        # do some forward ops
        return torch.sigmoid(torch.matmul(X, self.weight))

And my training code looks like this (The parameter self.weight is passed to the optimizer after I create the model):
def train(X_train, y_train):
    model = MyModel(y_train.shape[1])
    optimize = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = torch.nn.MSELoss()

    for epoch in range(10000):
        optimize.zero_grad()

        prediction = model(X_train)
        loss = loss_fn(prediction, y_train)
        loss.backward()

        optimize.step()

Output:
After all, it works for me using the way I explained in the comments - to allocate the weights parameter right in the init_parameters function.
import torch

class MyModel(torch.nn.Module):
    def __init__(self, out_dim):
        super(MyModel, self).__init__()
        self.weight = torch.nn.Parameter(torch.FloatTensor([0.0]))
        self.out_dim = out_dim
        self.init_weight = False

    def init_parameters(self, in_dim):
        self.weight = torch.nn.Parameter(torch.FloatTensor(in_dim, self.out_dim), requires_grad=True)
        torch.nn.init.xavier_normal_(self.weight)
        self.init_weight = True

    def forward(self, X):
        if not self.init_weight:
            # first call, so now I can initialize the weight since I know the input_dim
            self.init_parameters(X.shape[1])

        # do some forward ops
        result = torch.sigmoid(torch.matmul(X, self.weight))
        print(X.shape, result.shape)
        return result

def train(X_train, y_train):
    model = torch.nn.Sequential(MyModel(out_dim=100), MyModel(out_dim=20))
    optimize = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = torch.nn.MSELoss()

    for epoch in range(10000):
        #print('.', end='')
        optimize.zero_grad()

        prediction = model(X_train)
        loss = loss_fn(prediction, y_train)
        loss.backward()

        optimize.step()

batch_size, in_dim, out_dim = 100, 5, 20
X_train=torch.randn((batch_size, in_dim))
y_train=torch.randn((batch_size, out_dim))
train(X_train, y_train)


Instruction: torch suppress to kth largest values

Input:
I have the following function which works, but just not for half precision values (get a NotImplemented error for kthvalue).
def suppress_small_probabilities(probabilities: torch.FloatTensor, k: int) -> torch.FloatTensor:
    kth_largest, _ = (-probabilities).kthvalue(k, dim=-1, keepdim=True)
    return probabilities * (probabilities >= -kth_largest)

How would you do the equivalent without using kthvalue? I'm guessing topk has something to do with it, but I want to suppress the smaller values. probabilities is of size batch_size x 1000.

Output:
Implement your own topk, e.g.
def mytopk(xs: Tensor, k: int) -> Tensor:
    mask = torch.zeros_like(xs)
    batch_idx = torch.arange(0, len(xs))
    for _ in range(k):
        _, index = torch.where(mask == 0, xs, -1e4).max(-1)
        mask[(batch_idx, index)] = 1
    return mask

This will return a boolean mask tensor where the row-wise top-k elements will have value 1, rest 0.
Then use the mask to index your original tensor, e.g.
xs = torch.rand(3, 5, dtype=torch.float16)
# tensor([[0.0626, 0.9620, 0.5596, 0.4423, 0.1932],
#        [0.5289, 0.0857, 0.7802, 0.7730, 0.4807],
#        [0.8272, 0.5016, 0.1169, 0.4372, 0.1843]], dtype=torch.float16)
mask = mytopk(xs, 2)
# tensor([[0., 1., 1., 0., 0.],
#        [0., 0., 1., 1., 0.],
#        [1., 1., 0., 0., 0.]])
top_only = torch.where(mask == 1, xs, 0)
# tensor([[0.0000, 0.9620, 0.5596, 0.0000, 0.0000],
#        [0.0000, 0.0000, 0.7802, 0.7730, 0.0000],
#        [0.8271, 0.5016, 0.0000, 0.0000, 0.0000]], dtype=torch.float16)


Instruction: Writing a pytorch neural net class that has functions for both model fitting and prediction

Input:
I want a PyTorch neural net to predict y using x where, for example,
x = torch.tensor([[6,2],[5,2],[1,3],[7,6]]).float()
y = torch.tensor([1,5,2,5]).float()
For this, I have written the following PyTorch class which can fit y using x. But, I am not able to extend the code to predict using new values of x, x_test. Can somebody please guide me in this regard?
import torch
import torch.nn as nn
from torch.optim import SGD

class MyNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2,4,bias=True)
        self.layer2 = nn.Linear(4,1,bias=True)
        self.loss = nn.MSELoss()

    def fit(self,x,y):
        def forward(self,x):
            x = self.layer1(x)
            x = self.layer2(x)

            return x.squeeze()

        def compile(self,x):
            forward(self,x)
            opt = SGD(self.parameters(),lr=0.01) # parameters from first forward pass

        compile(self,x)

        losses = []
        for _ in range(100):
            opt.zero_grad() # flush previous epoch's gradient
            res = forward(self,x)
            loss_value = self.loss(res,y)
            loss_value.backward() # compute gradient
            opt.step() # Perform iteration using gradient above
            losses.append(loss_value.item())
        y_train_hat = forward(self,x)
        return y_train_hat.detach().numpy()

    def predict(self,x_test):

         """

         need help in this part

         """

        # y_test_hat = forward(self,x_test)
        # return y_test_hat.detach().numpy()

I changed the code provided by Ivan. The following is the working version of the code:
class MyNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2, 4, bias=True)
        self.layer2 = nn.Linear(4, 1, bias=True)
        self.loss = nn.MSELoss()
        self.compile_()

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return x.squeeze()

    def fit(self, x, y):

        losses = []
        for epoch in range(100):

            ## Inference
            res = self(x)  #instead of self(self,x)
            loss_value = self.loss(res,y)

            ## Backpropagation
            self.opt.zero_grad()  # flush previous epoch's gradient; this should be the first step, I think
            loss_value.backward() # compute gradient

            self.opt.step()       # Perform iteration using gradient above

            ## Logging
            losses.append(loss_value.item())

    def compile_(self):
        self.opt = SGD(self.parameters(), lr=0.01)

    def predict(self,x_test):
        self.eval()
        y_test_hat = self(x_test) #instead of forward(self,x_test)
        return y_test_hat.detach().numpy()
        self.train()

x = torch.tensor([[6,2],[5,2],[1,3],[7,6]]).float()
y = torch.tensor([1,5,2,5]).float()

model = MyNeuralNet()
model.fit(x,y)
model.predict(x)

Is this correct?

Output:
Here is a minimal example with some modifications carried out to your code:
class MyNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2, 4, bias=True)
        self.layer2 = nn.Linear(4, 1, bias=True)
        self.loss = nn.MSELoss()
        self.compile_()

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return x.squeeze()

    def fit(self, x, y):
        losses = []
        for epoch in range(100):
            ## Inference
            res = self(self,x)
            loss_value = self.loss(res,y)

            ## Backpropagation
            loss_value.backward() # compute gradient
            self.opt.zero_grad()  # flush previous epoch's gradient
            self.opt.step()       # Perform iteration using gradient above

            ## Logging
            losses.append(loss_value.item())

    def compile_(self, x):
        self.opt = SGD(self.parameters(), lr=0.01)

    def predict(self, x_test):
        self.eval()
        y_test_hat = self(x_test)
        return y_test_hat.detach().numpy()
        self.train()

Which can be used like this for training:
>>> model = MyNeuralNet()
>>> model.fit(x, y) # fit on some data (x, y)

Then perform inference with some other data x_test:
>>> model.predict(x_test)


Instruction: How to feed different pad IDs to a collate function?

Input:
I usually use a custom collate_fn and use it as an argument when defining my DataLoader. It usually looks something like:
def collate_fn(batch):
    max_len = max([len(b['input_ids']) for b in batch])
    input_ids = [b['input_ids'] + ([0] * (max_len - len(b['input_ids'])))]
    labels = [b['label'] for b in batch]
    return input_ids

As you can see, I'm using 0 for my padding sequence. What I'm wondering is, since language models and their tokenizers use different IDs for padding tokens, is there a way that I can make the collate_fn flexible to take that into account?

Output:
I was able to make a workaround by making a Trainer class and making the collate_fn a method. After that I was able to do something like self.pad_token_id = tokenizer.pad_token_id and modify the original collate_fn to use self.pad_token_id rather than a hardcoded value.
I'm still curious if there's any way to do this while keeping collate_fn a top-level function though. For example if there would be any way to pass an argument or something.
<Original>

def collate_fn(batch):
    max_len = max([len(b['input_ids']) for b in batch])
    input_ids = [b['input_ids'] + ([0] * (max_len - len(b['input_ids']))) for b in batch]
    return input_ids

class Trainer():
    def __init__(self, tokenizer, ...):
        ...

    def train(self):
        train_dataloader = DataLoader(features, collate_fn=collate_fn, ...)

    ...

<Workaround>
class Trainer():
    def __init__(self, tokenizer, ...):
        self.pad_token_id = tokenizer.pad_token_id
        ...

    def collate_fn(self, batch):
        max_len = max([len(b['input_ids']) for b in batch])
        input_ids = [b['input_ids'] + ([self.pad_token_id] * (max_len - len(b['input_ids']))) for b in batch]
        return input_ids

    def train(self):
        train_dataloader = DataLoader(features, collate_fn=self.collate_fn, ...)

    ...


Instruction: Ensure that every column in a matrix has at least `e` non-zero elements

Input:
I would like to ensure that each column in a matrix has at least e non-zero elements, and for each column that does not randomoly replace zero-valued elements with the value y until the column contains e non-zero elements. Consider the following matrix where some columns have 0, 1 or 2 elements. After the operation, each column should have at least e elements of value y.
before
tensor([[0, 7, 0, 0],
        [0, 0, 0, 0],
        [0, 1, 0, 4]], dtype=torch.int32)

after, e = 2
tensor([[y, 7, 0, y],
        [y, 0, y, 0],
        [0, 1, y, 4]], dtype=torch.int32)

I have a very slow and naive loop-based solution that works:
def scatter_elements(x, e, y):
  for i in range(x.shape[1]):
    col = x.data[:, i]
    num_connections = col.count_nonzero()
    to_add = torch.clip(e - num_connections, 0, None)
    indices = torch.where(col == 0)[0]
    perm = torch.randperm(indices.shape[0])[:to_add]
    col.data[indices[perm]] = y

Is it possible to do this without loops? I've thought about using torch.scatter and generate an index array first, but since the number of elements to be added varies per column, I see no straightforward way to use it. Any suggestions or hints would be greatly appreciated!
Edit: swapped indices and updated title and description based on comment.

Output:
In the case where you care only that each column has at least e elements and not EXACTLY e elements, you can do it without a loop. The key is that in this case, we can create an array with every non-zero value replaced, and then sample e values from this array for each column.
For convenience let x.shape = [a,b]

Create an array replace with every value replaced (i.e. every 0 replaced with y).
Create a random array of same size as x.
Use torch.topk to get the k largest random numbers per column. This is used to get k random indices for each column (in your case k = e). Provided that x is non-negative integer, you can add x to the random array before the topk operation to ensure that the existing non-zero elements are selected first; this ensures that no more than e connections are added.
Replace the indexed values per row with the values from replace.

def scatter_elements(x,e,y):
  x = x.float()
  # 1. replace has same shape as x and has all 0s replaced with y
  replace = torch.where(x > 0 , x, torch.ones(x.shape)*y)

  # 2-3. get random indices per column
  randn = torch.rand(x.shape)
  if True: # True if you don't want the modification to ever itself assign more than e elements in a column a non-zero value
      randn += x # assumes x is non-negative integer

  ind = torch.topk(randn,e,dim = 0)[1] # first return is values, second return is indices

  # create a second index to indicate which column each index in ind corresponds to
  col_ind = torch.arange(x.shape[1]).unsqueeze(0).expand(ind.shape)

  # 4. Index x with ind and col_ind and set these values to the corresponding values in replace
  ind = ind.reshape(-1)         # flatten into 1D array so we can use to index row
  col_ind = col_ind.reshape(-1) # flatten into 1D array so we can use to index column
  x[ind,col_ind] = replace[ind,col_ind]

  return x

In my limited timing tests, the vectorized solution was about 5-6x faster than the original looping solution.


Instruction: Multiply each tensor with a value from a another tensor

Input:
I have two tensors:
import torch
a = torch.randn((2,3,5))
b = torch.tensor([[2.0, 1.0, 2.0],[0.5, 1.0, 1.0]])

And I want to multiply the each element in the last dimension in a with the corresponding element in b. That means when a is:
tensor([[[ 1, 2, 3, 4,  5],
     [1, 2,  3,  4,  5],
     [1, 2,  3,  4,  5]],
    [[1, 2,  3,  4,  5],
     [1, 2,  3,  4,  5],
     [1, 2,  3,  4,  5]]])

the result should be:
tensor([[[ 2, 4, 6, 8,  10],
     [1, 2,  3,  4,  5],
     [ 2, 4, 6, 8,  10]],
    [[0.5, 1.0,  1.5,  2.0,  2.5],
     [1, 2,  3,  4,  5],
     [1, 2,  3,  4,  5]]])

How can I do that?

Output:
All I need to do, is to add an dimension:
a * b.unsqueeze(-1)


Instruction: Giving less weight to data coming from another dataset that is noisy

Input:
I have two datasets, one with clean data and one with dirty data. I train a Roberta model on the clean dataset and then get predictions for the dirty dataset. Those predictions with a probability greater than 0.9 go to the clean dataset. I then retrain the Roberta model with this new dataset (clean + dirty moving to clean).
For the retraining I am using the MAE loss function (more robust to noisy labels) and I use weights to give less value to the data that passes from the dirty to the clean dataset, as follows:
loss = torch.mean(torch.abs(y_true - y_pred) * weights)

Initially I am using an arbitrary weight of 0.5 for all the dirty data that gets passed into the clean dataset. However, I would like to assign them a weight in a more academic way, not so arbitrary.
How can I do that?

Output:
One way to choose the weight is based on your confidence in the dirty data and assign the weight accordingly. For example, if you think that 90% of dirty data is labeled correctly, then choosing 0.9 as the weight for the noisy data is a reasonable option.
Additionally, there is a whole literature on learning from noisy labels, you can check this survey for more information: https://arxiv.org/abs/2007.08199


Instruction: ValueError: too many values to unpack using sum return function

Input:
Attempting to implement this Deep Embedded Clustering GitHub algorithm.
    def acc(y_pred, y_target):
        D = max(y_pred.max(), y_target.max()) + 1
        w = np.zeros((D, D), dtype=np.int64)
        for i in range(y_pred.size):
            w[y_pred[i], y_target[i]] += 1

        ind = linear_assignment(w.max() - w)
        return sum(w[i, j] for i, j in ind) * 1.0 / y_pred.size # <- Error Line

Just using the code within the repo I am encountering this ValueError. I attempted to use the zip function to solve this as well as assigning the output to a variable before passing return.

Output:
Solution if anyone else encounters similar issue. The Github Repo using linear_assignment  is deprecated and removed from updated scikit learn packages.
I had used the solution accepted as the answer within this thread , However, as mentioned in that thread scipy.optimize.linear_sum_assignment is not the perfect replacement for linear assignment. This was causing my issue as the output of the two functions is different.
Two options is you can downgrade your Scikit learn package to the version which supports linear assignment, or comment out the import and use the old function definition as posted by InputBlackBoxOutput in linked thread, and below.
def linear_assignment(cost_matrix):
  try:
    import lap
    _, x, y = lap.lapjv(cost_matrix, extend_cost=True)
    return np.array([[y[i], i] for i in x if i >= 0])
  except ImportError:
    from scipy.optimize import linear_sum_assignment
    x, y = linear_sum_assignment(cost_matrix)
    return np.array(list(zip(x, y)))


Instruction: Defining my own gradient function for pytorch to use

Input:
I want to feed pytorch gradients manually. In my real problem, I have my own adjoint function that does not use tensors. Is there any way I can define my own gradient function for pytorch to use during optimization?
import numpy as np
import torch

# define rosenbrock function and gradient
x0 = np.array([0.1, 0.1])
a = 1
b = 5
def f(x):
   return (a - x[0]) ** 2 + b * (x[1] - x[0] ** 2) ** 2

def jac(x):
   dx1 = -2 * a + 4 * b * x[0] ** 3 - 4 * b * x[0] * x[1] + 2 * x[0]
   dx2 = 2 * b * (x[1] - x[0] ** 2)
   return np.array([dx1, dx2])

# create stochastic rosenbrock function and gradient
# (the crude analogy is that I have predefined stochastic
#  forward and backward functions)
def f_rand(x):
   return f(x) * np.random.uniform(0.5, 1.5)

def jac_rand(x): return jac(x) * np.random.uniform(0.5, 1.5)

x_tensor = torch.tensor(x0, requires_grad=False)
optimizer = torch.optim.Adam([x_tensor], lr=0.1)

# here, closure is fed f_rand to compute the gradient.
# I need to feed closer the gradient directly from jac_rand
def closure():
   optimizer.zero_grad()
   loss = f_rand(x_tensor)
   loss.backward() # jac_rand(x)
   return loss

for ii in range(200):
   optimizer.step(closure)

print(x_tensor, f(x_tensor))
  # tensor([1.0000, 1.0000], dtype=torch.float64, requires_grad=True) tensor(4.5799e-09, dtype=torch.float64, grad_fn=<AddBackward0>)
  # ( this is the right answer, E[f(1, 1)] = 0 )

I've tried defining a custom function, but I can't get it to work. This is my best attempt so far:
import numpy as np
import torch

# define rosenbrock function and gradient

x0 = np.array([0.1, 0.1])
a = 1
b = 5
def f(x):
   return (a - x[0]) ** 2 + b * (x[1] - x[0] ** 2) ** 2

def jac(x):
   dx1 = -2 * a + 4 * b * x[0] ** 3 - 4 * b * x[0] * x[1] + 2 * x[0]
   dx2 = 2 * b * (x[1] - x[0] ** 2)
   return np.array([dx1, dx2])

# create stochastic rosenbrock function and gradient
def f_rand(x):
   return f(x) * np.random.uniform(0.5, 1.5)

def jac_rand(x): return jac(x) * np.random.uniform(0.5, 1.5)

class custom_function(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input):
       ctx.save_for_backward(input)
       return f_rand(input)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        return grad_output * g_rand(input)

x_tensor = torch.tensor(x0, requires_grad=False)
optimizer = torch.optim.Adam([x_tensor], lr=0.1)

for ii in range(200):
   print('x_tensor ', x_tensor)
   optimizer.step(custom_function())

print(x_tensor, f(x_tensor))

It says:
RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)

Output:
Not quite sure if this is exactly what you want but the method call loss.backward() computes gradients via pytorch's computational graph and stores the gradient values in the weight tensors themselves (in your case it's in x_tensor). And these gradients can be accessed via x_tensor.grad. However, if you don't want to use pytorch's gradient computing method using loss.backward(), then you can manually feed your gradients into your tensor's .grad attribute as follows:
with torch.no_grad():
    def closure():
       optimizer.zero_grad()
       loss = f_rand(x_tensor)
       x_tensor.grad = torch.from_numpy(jac_rand(x_tensor))
       return loss


Instruction: Where to find the docs of axis parameter of torch.sum?

Input:
I'm trying to read about the sum function of torch here. I noticed that the following works:
> print(torch.sum(torch.randint(0,2,(2,2)),axis=1))
tensor([1, 0])

But in the docs above I don't see explanation for axis.
In the signature of the function I see *:
torch.sum(input, *, dtype=None) → Tensor

Does it have to do something with this *? Where can I find the PyTorch docs that explains how to use axis? I came across with the same thing with other methods (with other arguments). So, although I know how to use axis, I would want to figure how to actually read the docs for the "hidden" arguments.

Output:
PyTorch emulates much of the basic functionality of Numpy (with additional GPU acceleration and autograd mechanics) but the API also differs in some small ways. For example, PyTorch generally uses the dim keyword argument to specify which dimension a function should operate on while Numpy uses the axis keyword argument to specify the same thing.
While undocumented it appears that the PyTorch devs have allowed users to use the axis keyword argument in place of dim for the sum function. This is probably to make the library more compatible with NumPy functions.
Therefore
torch.sum(x, dim=1)

is equivalent to
torch.sum(x, axis=1)

and both specify that you want to sum-reduce along dimension 1.
You can read about the NumPy version of sum here, which also provides additional examples of the axis argument.


Instruction: Adapting MNIST designed network for a larger dataset

Input:
I am attempting to implement this deep clustering algorithm, which was designed to cluster the MNIST dataset. Single Channel 28x28 images.
The images I am trying to use are 416x416 and 3-Channel RGB. The script is initialised with the following functions.
class CachedMNIST(Dataset):
    def __init__(self, train, cuda, testing_mode=False):
        img_transform = transforms.Compose([transforms.Lambda(self._transformation)])
        # img_transform = transforms.Compose([transforms.Resize((28*28)), transforms.ToTensor(), transforms.Grayscale()])
        self.ds = torchvision.datasets.ImageFolder(root=train, transform=img_transform)
        self.cuda = cuda
        self.testing_mode = testing_mode
        self._cache = dict()

    @staticmethod
    def _transformation(img):
        return (torch.ByteTensor(torch.ByteStorage.from_buffer(img.tobytes())).float()
                * 0.02
        )

If the images are left un-altered the resulting tensor shape output from the _transformation function is of size torch.Size{{256,519168]] far too large for the AutoEncoder network to calculate.
Error 1
RuntimeError: mat1 and mat2 shapes cannot be multiplied (128x519168 and 784x500)
When I attempted to resize the images a the result is a 4D Tensor, torch.Size([256,1,784,748]) which even when reducing the Batch Size to minuscule amounts the CUDA will crash as there is not enough memory.
Error 2
RuntimeError: CUDA out of memory.
I'm hoping someone can point me in the right direction to tackle this problem as there must be a more efficient way to adapt the network.
AutoEnocder Model
StackedDenoisingAutoEncoder(
  (encoder): Sequential(
    (0): Sequential(
      (linear): Linear(in_features=784, out_features=500, bias=True)
      (activation): ReLU()
    )
    (1): Sequential(
      (linear): Linear(in_features=500, out_features=500, bias=True)
      (activation): ReLU()
    )
    (2): Sequential(
      (linear): Linear(in_features=500, out_features=2000, bias=True)
      (activation): ReLU()
    )
    (3): Sequential(
      (linear): Linear(in_features=2000, out_features=10, bias=True)
    )
  )
  (decoder): Sequential(
    (0): Sequential(
      (linear): Linear(in_features=10, out_features=2000, bias=True)
      (activation): ReLU()
    )
    (1): Sequential(
      (linear): Linear(in_features=2000, out_features=500, bias=True)
      (activation): ReLU()
    )
    (2): Sequential(
      (linear): Linear(in_features=500, out_features=500, bias=True)
      (activation): ReLU()
    )
    (3): Sequential(
      (linear): Linear(in_features=500, out_features=784, bias=True)
    )
  )
)

Output:
Error 1 is happening because the first linear layer has in_features=784.  That number comes from the 28x28 pixels in the 1-channel MNIST data.  Your input data is 416x416x3 = 519168 (different if you resize your inputs).  In order to resolve this error, you need to make the in_features in that first linear layer match the number of pixels (times the number of channels) of your input.  You can do this by changing that number or resizing your input (or, likely both).  Also, note that you will likely have to flatten your input so that it is a vector.  Also, note that whatever the in_features becomes (to the encoder) you'll want to make the out_features of the decoder match (otherwise you'll be trying to compare two vectors of different sizes when training).
Error 2 CUDA OOM could happen for lots of reasons (small GPU, too large of network, too large batch size, etc).  The network you have doesn't appear to look particularly large.  But you could reduce its size by shrinking some of the internal layers (numbers of in_features and out_features).  Just be sure that if you adjust these, that you maintain the property that the number of out_features from one layer matches the number of in_features on the next layer.  And, in this example, the decoder is a nice mirror of in the encoder (so if you adjust the encoder, make the corresponding mirror-adjustment in the decoder).


Instruction: Pytorch "hipErrorNoBinaryForGpu: Unable to find code object for all current devices!"

Input:
SYSTEM: Ryzen 5800x, rx 6700xt, 32 gigs of RAM, Ubuntu 22.04.1
I'm attempting to install Stable-Diffusion by following https://youtu.be/d_CgaHyA_n4
When attempting to run the SD script, I get the "hipErrorNoBinaryForGpu: Unable to find code object for all current devices!" error.
I believe this is caused by PyTorch not working as expected. When validating Pytorchs' installation with "The Master Test", I get the same error:
"hipErrorNoBinaryForGpu: Unable to find code object for all current devices!"
Aborted (core dumped)

I believe that it is install correctly as using the conda list command tells me that torch 1.12.0a0+git2a932eb and torchvision 0.13.0a0+f5afae5 are installed. Interestingly, when I change the command ever-so-slightly to torch.cuda.is_available, (without the parentheses), I get the following output: <function is_available at 0x7f42278788b0>. Granted, I'm not sure what this is telling me. Following the "Verification" step resulted in the expected array of random numbers. However, failed the GPU driver check.
Thank you in advance.

Output:
Try running the following command:
export HSA_OVERRIDE_GFX_VERSION=10.3.0
This made it work on my machine using an RX 6600 XT, with which I got the same error running it, before exporting the variable.


Instruction: IterableWrapper is not defined when using WikiText2

Input:
I am trying to follow along this tutorial https://pytorch.org/tutorials/beginner/transformer_tutorial.html
I am getting the following error when calling this function.
----> 6 train_iter = WikiText2(split='train')
/usr/local/lib/python3.7/dist-packages/torchtext/datasets/wikitext2.py in WikiText2(root, split)
75         )
76
---> 77     url_dp = IterableWrapper([URL])
78     # cache data on-disk
79     cache_compressed_dp = url_dp.on_disk_cache(
NameError: name 'IterableWrapper' is not defined
Here is the code:
from torchtext.datasets import WikiText2
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator
from torchdata.datapipes.iter import IterableWrapper

train_iter = WikiText2(split='train')

Let me know if you have any ideas.. Thanks
Antoine

Output:
I tried running the snippet of code you provided. I don't see
NameError: name 'IterableWrapper' is not defined
but I have a different error which says,
No module named 'torchdata'
I don't have torchdata installed.
So in your case, I would make sure if the torchdata is installed correctly.
You can look at this official Git and see if it works out
https://github.com/pytorch/data
Best regards


Instruction: Pytorch model output is not correct (torch.float32 and torch.float64)

Input:
I have created a DNN model with Pytorch (input_dim=6, output_dim=150). Normally, if I generate a random X_in=torch.randn(6000, 6), it will return me a model_out.shape=(6000, 150), and if I calculate the Rank of model_out, it should be 150 (since my model's weight and bias are also randomly initialised).
However, you can see this is NOT TRUE with the following code:
import torch
import torch.nn as nn

torch.manual_seed(923) # for reproducible result

class MyDNN(nn.Module):
    def __init__(self):
        super(MyDNN, self).__init__()
        # layer 0:
        self.linear_0 = nn.Linear(6, 150)
        self.activ_0 = nn.Tanh()
        # layer 1:
        self.linear_1 = nn.Linear(150, 150)
        self.activ_1 = nn.Tanh()
        # layer 2:
        self.linear_2 = nn.Linear(150, 150)
        self.activ_2 = nn.Tanh()
        # layer 3:
        self.linear_3 = nn.Linear(150, 150)
        self.activ_3 = nn.Tanh()

    def forward(self, x):
        out = self.activ_0(self.linear_0(x)) # output: layer 0
        out = self.activ_1(self.linear_1(out)) # output: layer 1
        out = self.activ_2(self.linear_2(out)) # output: layer 2
        out = self.activ_3(self.linear_3(out)) # output: layer 3
        return out

model = MyDNN()
X_in = torch.randn(6000, 6, dtype=torch.float32)
with torch.no_grad():
    model_out = model(X_in)
print(f'model_out rank = {torch.linalg.matrix_rank(model_out)}')

model_out rank = 115. Apparently this is a WRONG output, there is no way that the output has so many linear dependent columns with all the inputs, weights and bias are randomly initialised!
This problem can be solved by changing the X_in dtype as well as the model dtype to float64 with the following code:
model_64 = MyDNN()
model_64.double()
X_in_64 = torch.randn(6000, 6, dtype=torch.float64)
with torch.no_grad():
    model_64_out = model_64(X_in_64)
print(f'model_64_out rank = {torch.linalg.matrix_rank(model_64_out)}')

model_64_out rank = 150
Here is my question:

Why does this happen? Is this really a problem of data size? I mean float32 already has a good precision. Actually when I use my own training_data, even with mini_batch_size = 10 -> output.shape = (10, 150), my Rank(output) is less than 10.
Although this problem can be solved by using double precision, this slows down the whole training process a lot (and with Mac M1 pro GPU, it only supports float32 type). Is there any other solution?

Output:
You have to realize that we are dealing with a numerical problem here: The rank of a matrix is a discrete value derived from a e.g. a singular value decomposition in the case of torch.matrix_rank. In this case we need to consider a threshold on the singular values: At what modulus tol do we consider a singular value as exactly zero?
Remember that we are dealing with floating point values where all operations always comes with truncation and rounding errors. In short there is no sense in trying to compute an exact rank.
So instead you might reconsider what kind of tolerance you use, you could e.g. use torch.linal.matrix_rank(..., tol=1e-6). The smaller the tolerance, the higher the expected rank.
But no matter what kind of floating point precision you use, I'd argue you will never be able to find meaningful "exact" number for the rank, it will always be a trade off! Therefore I'd reconsider whether you really need to compute the rank in the first place, or wether there is some other kind of criterion that is better suited for numerical considerations in the first place!