Instruction: Import error while launching PyTorch Lightning project on Colab TPU

Input:
I followed this guide to launch my PyTorch Lightning project on Google Colab TPU. So I installed
!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.9-cp37-cp37m-linux_x86_64.whl

Then
 !pip install pytorch-lightning

Then I
!pip install torch torchvision torchaudio
!pip install -r requirements.txt

After installing the project requirements, I restarted the runtime as requested and re-ran the cloud-TPU-client install, the pytorch-lightning install, and both command from above. It ran smoothly.
But just after the TPU has started with version PyTorch version 1.9, I get the following import error :
WARNING:root:TPU has started up successfully with version pytorch-1.9
        Traceback (most recent call last):
          File "synthesizer_train.py", line 2, in <module>
            from synthesizer.train import train
          File "/content/Real-Time-Voice-Cloning/synthesizer/train.py", line 6, in <module>
            from synthesizer.models.tacotron import Tacotron
          File "/content/Real-Time-Voice-Cloning/synthesizer/models/tacotron.py", line 7, in <module>
            import pytorch_lightning as pl
          File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/__init__.py", line 20, in <module>
            from pytorch_lightning.callbacks import Callback  # noqa: E402
          File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
            from pytorch_lightning.callbacks.base import Callback
          File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/callbacks/base.py", line 26, in <module>
            from pytorch_lightning.utilities.types import STEP_OUTPUT
          File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/__init__.py", line 18, in <module>
            from pytorch_lightning.utilities.apply_func import move_data_to_device  # noqa: F401
          File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/apply_func.py", line 26, in <module>
            from pytorch_lightning.utilities.imports import _compare_version, _TORCHTEXT_AVAILABLE
          File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/imports.py", line 101, in <module>
            from pytorch_lightning.utilities.xla_device import XLADeviceUtils  # noqa: E402
          File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/xla_device.py", line 24, in <module>
            import torch_xla.core.xla_model as xm
          File "/usr/local/lib/python3.7/dist-packages/torch_xla/__init__.py", line 142, in <module>
            import _XLAC
        ImportError: /usr/local/lib/python3.7/dist-packages/_XLAC.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN2at13_foreach_erf_EN3c108ArrayRefINS_6TensorEEE

Trainer was launched with the flag TPU_cores=8.
The model had run on CPU and GPU beforehand (ie on another session).
I tried to downgrade PyTorch to 1.9 (the same as the one shown when TPU is starting) because Colab uses torch 1.10.0+cu111 and a different error appeared :
WARNING:root:TPU has started up successfully with version pytorch-1.9
Traceback (most recent call last):
  File "synthesizer_train.py", line 2, in <module>
    from synthesizer.train import train
  File "/content/Real-Time-Voice-Cloning/synthesizer/train.py", line 6, in <module>
    from synthesizer.models.tacotron import Tacotron
  File "/content/Real-Time-Voice-Cloning/synthesizer/models/tacotron.py", line 7, in <module>
    import pytorch_lightning as pl
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/__init__.py", line 20, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.base import Callback
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/callbacks/base.py", line 26, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/__init__.py", line 18, in <module>
    from pytorch_lightning.utilities.apply_func import move_data_to_device  # noqa: F401
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/apply_func.py", line 29, in <module>
    if _compare_version("torchtext", operator.ge, "0.9.0"):
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/imports.py", line 54, in _compare_version
    pkg = importlib.import_module(package)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.7/dist-packages/torchtext/__init__.py", line 5, in <module>
    from . import vocab
  File "/usr/local/lib/python3.7/dist-packages/torchtext/vocab/__init__.py", line 11, in <module>
    from .vocab_factory import (
  File "/usr/local/lib/python3.7/dist-packages/torchtext/vocab/vocab_factory.py", line 4, in <module>
    from torchtext._torchtext import (
ImportError: /usr/local/lib/python3.7/dist-packages/torchtext/_torchtext.so: undefined symbol: _ZTVN5torch3jit6MethodE

Is there anything I can do to train the model on TPU ?
Thank you very much

Output:
Actually the same problem has also been described and the suggested solution did work for me.
So in the details they suggest to downgrade PyTorch to 1.9.0+cu111 (mind the +cu111) after installing torch_xla.
Consequently here are the steps I followed to launch my Lightning project on Google Colab with TPU :
!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.9-cp37-cp37m-linux_x86_64.whl
!pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchtext==0.10.0 -f https://download.pytorch.org/whl/cu111/torch_stable.html

And then the project's pip :
!pip install torch torchvision torchaudio pytorch-lightning
!pip install -r requirements.txt

And it worked even though after this last step, I had to restart runtime.


Instruction: WSL2 Pytorch - RuntimeError: No CUDA GPUs are available with RTX3080

Input:
I have been struggling for day to make torch work on WSL2 using an RTX 3080.
I Installed the CUDA-toolkit version 11.3
Running nvcc -V returns this :
nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0

nvidia-smi returns this
Mon Nov 29 00:38:26 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.00       Driver Version: 510.06       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  N/A |
| N/A   52C    P5    17W /  N/A |   1082MiB / 16384MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I verified the installation of the toolkit with blackscholes
./BlackScholes
[./BlackScholes] - Starting...
GPU Device 0: "Ampere" with compute capability 8.6

Initializing data...
...allocating CPU memory for options.
...allocating GPU memory for options.
...generating input data in CPU mem.
...copying input data to GPU mem.
Data init done.

Executing Black-Scholes GPU kernel (512 iterations)...
Options count             : 8000000
BlackScholesGPU() time    : 0.242822 msec
Effective memory bandwidth: 329.459087 GB/s
Gigaoptions per second    : 32.945909

BlackScholes, Throughput = 32.9459 GOptions/s, Time = 0.00024 s, Size = 8000000 options, NumDevsUsed = 1, Workgroup = 128

Reading back GPU results...
Checking the results...
...running CPU calculations.

Comparing the results...
L1 norm: 1.741792E-07
Max absolute error: 1.192093E-05

Shutting down...
...releasing GPU memory.
...releasing CPU memory.
Shutdown done.

[BlackScholes] - Test Summary

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Test passed

And when I try to use torch, it doesn't find any GPU. Btw, I had to install torch==1.10.0+cu113 if I wanted to use torch with my RTX 3080 as the sm_ with the simple 1.10.0 version are not compatible with the rtx3080.
Running torch returns this :
>>> import torch
>>> torch.version
<module 'torch.version' from '/home/snihar/miniconda3/envs/tscbasset/lib/python3.7/site-packages/torch/version.py'>
>>> torch.version.cuda
'11.3'
>>> torch.cuda.get_arch_list()
[]
>>> torch.cuda.device_count()
0
>>>  torch.cuda.current_device()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/snihar/miniconda3/envs/tscbasset/lib/python3.7/site-packages/torch/cuda/__init__.py", line 479, in current_device
    _lazy_init()
  File "/home/snihar/miniconda3/envs/tscbasset/lib/python3.7/site-packages/torch/cuda/__init__.py", line 214, in _lazy_init
    torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available

At last, interestingly, I am completely able to run tensorflow-gpu on the same machine.
Installed pytorch like this : conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
Also, I managed to run pytorch in a docker container started from my WSL2 machine with this command :
sudo docker run --gpus all -it --rm -v /home/...:/home/... nvcr.io/nvidia/pytorch:21.11-py3.

When running pytorch on the windows machine I am running the WSL from, it works too. Both return  ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37'] which says that the library is compatible with rtx 3080.

Output:
In my case, I solved this issue by linking /usr/lib/wsl/lib/libcuda.so.1 to the libcuda.so in your wsl2 CUDA location. See https://github.com/microsoft/WSL/issues/5663
After reboot, pytorch can find the GPU.
(I found the warning " /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link" during the apt-get upgrade command. Not sure you can solve it in the same way) Downgrade to pytorch 1.8.2LTS can also solve the problem, but the calculation speed is extremely low.


Instruction: Counting number of occurrences in PyTorch Tensor. (Tensor is too big for Numpy)

Input:
Is there any smart way to count the number of occurrences of each value in a very Large PyTorch Tensor? Tensor Size is 11701*300=3510300 or maybe increase or decrease.
TORCH.BINCOUNT, TORCH.UNIQUE and TORCH.UNIQUE_CONSECUTIVE are not useful so far.
BINCOUNT returns a different number of elements every time. Unique is also not useful as it returns unique values.

This is what I meant when I said it returns different elements every time. If 5 elements will return 8 elements tensor, How I am supposed to know which elements are how many times. this is confusing for me. The official documentation has limited content and there is no other website, explains it.
In the above picture. So, 5 is 2 times. 0 is? what 0 times? How to read this output. it doesn't make any sense to me.

Output:
Actually the problem is how you read the output. The output of torch.bincount is a tensor of size max(input)+1, that means it covers all bins of size 1 from zero to max(input). Therefore, in the output tensor from the first element you see how many 0, 1, 2, ..., max(input) are there in your non-negative integral array.
For example:
t1 = torch.randint(0,10, (20,))
print(t1)

tensor([2, 5, 7, 3, 1, 2, 7, 8, 8, 0, 5, 6, 4, 4, 4, 6, 3, 0, 6, 6])

in this tensor the max value is 8 (9 did not appear by chance), so it gives:
print(torch.bincount(t1).size())
print(torch.bincount(t1))

torch.Size([9])
tensor([2, 1, 2, 2, 3, 2, 4, 2, 2])

that means, in the tensor t1 there are two 0s, one 1, two 3s, ..., and two 8s.


Instruction: How to clean garbage from CUDA in Pytorch?

Input:
I teached my neural nets and realized that even after torch.cuda.empty_cache() and gc.collect() my cuda-device memory is filled. In Colab Notebooks we can see the current variables in memory, but even I delete every variable and clean the garbage gpu-memory is busy.
I heard it's because python garbage collector can't work on cuda-device.
Please explain me, what should I do?

Output:
You can do this:
import gc
import torch
gc.collect()
torch.cuda.empty_cache()


Instruction: How do you prevent the tensorboard logger in pytorch lightning from logging the current epoch?

Input:
When creating a new tensorboard logger in pytorch lightning, the two things that are logged by default are the current epoch and the hp_metric. I was able to disable the hp_metric logging by setting default_hp_metric=False but I can't find anything to disable the logging of the epoch. I've searched in the lightning.py, trainer.py and tensorboard.py files which have the code for the module, the trainer and the tensorboard logger and couldn't find a logging call for epoch anywhere.
This behavior occurs even taking the barebones example from the pytorch lightning tutorial.
Is there a way to disable this logging of epoch to prevent clutter in the tensorboard interface?

Output:
In Short
You can disable automatically writing epoch variable by overwriting tensorboard logger.
from pytorch_lightning import loggers
from pytorch_lightning.utilities import rank_zero_only

class TBLogger(loggers.TensorBoardLogger):
    @rank_zero_only
    def log_metrics(self, metrics, step):
        metrics.pop('epoch', None)
        return super().log_metrics(metrics, step)

Full version

Pytorch lightning automatically add epoch vs global_step graph to each logger. (you can see description in here)
There is no option to turn this behavior off. Because this is hard coded without any condition like below: (see full source code in here)
if step is None:
    # added metrics for convenience
    scalar_metrics.setdefault("epoch", self.trainer.current_epoch)
    step = self.trainer.global_step

# log actual metrics
self.trainer.logger.agg_and_log_metrics(scalar_metrics, step=step)

To disable this option, you should pop epoch variable from metric dictionary in log_metrics(metrics, step) that is called in add_and_log_metrics(scalar_metrics, step=step). Code is shown in above. You can see full long version snippet in here.


Instruction: How can I get labels from data generator?

Input:
I try to generate data for my conditional VAE and I need labels, but after generating data when I want to get labels I got this error:
def gen_batch(BATCH_SIZE):
    labels = torch.randint(0, 8, (BATCH_SIZE,)).long().to(device)
    theta = (np.pi/4) * labels.float().to(device)
    centers = torch.stack((torch.cos(theta), torch.sin(theta)), dim = -1)
    noise = torch.randn_like(centers) * 0.1

    return centers + noise, labels

def data_gen(BATCH_SIZE):
    #8 gaussians
    while 1:
        yield gen_batch(BATCH_SIZE)
train_loader,train_labels = data_gen(args.batch_size)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-17-3bc469737639> in <module>
----> 1 train_loader,train_labels = data_gen(args.batch_size)

ValueError: too many values to unpack (expected 2)

how can I fix that?

Output:
The exception was raised, because the data_gen() is a generator function, and such a function returns only a single object called a generator object, which cannot be unpacked like you tried to do. A generator object is an iterator that generates and returns data on demand and for that iterators support a special method called __next__, that returns only a single element at a time. However, you do not usually call this method directly. You either pass an iterator to the next built-in function, or you use it, for example, in a for loop that will call next() behind the scenes. Also, the important thing to remember is that after you have exhausted an iterator, you cannot use it again, and if you want to do that, you simply need to create a new one. In your case, though, you cannot exhaust your iterator, because you have an infinite while loop in the data_gen function. Here's an example:
def data_gen(BATCH_SIZE):
    #8 gaussians
    while 1:
        yield gen_batch(BATCH_SIZE)

gen_obj = data_gen(args.batch_size)
train_loader, train_labels = next(gen_obj)
# or:
gen_obj = data_gen(args.batch_size)
for train_loader, train_labels in gen_obj:
    # this is only an example and the for loop
    # will never end, because in
    # your case the generator is infinite
    pass


Instruction: How to easily convert a PyTorch dataloader to tf.Dataset?

Input:
How can we convert a pytorch dataloader to a tf.Dataset?
I spied this snippet:-
def convert_pytorch_dataloader_to_tf_dataset(dataloader, batch_size, shuffle=True):
    dataset = tf.data.Dataset.from_generator(
        lambda: dataloader,
        output_types=(tf.float32, tf.float32),
        output_shapes=(tf.TensorShape([256, 512]), tf.TensorShape([2,]))
    )
    if shuffle:
        dataset = dataset.shuffle(buffer_size=len(dataloader.dataset))
    dataset = dataset.batch(batch_size)
    return dataset

But it doesn't work at all.
Is there an in-built option to export dataloaders to tf.Datasets easily? I have a very complex dataloader, so a simple solutions should ensure things are bug-free :)

Output:
For your data in h5py format, you can use the script below. name_x is the features' name in your h5py and name_y is your label's file name. This method is memory efficient and you can feed the data batch by batch.
class Generator(object):

def __init__(self,open_directory,batch_size,name_x,name_y):

    self.open_directory = open_directory

    data_f = h5py.File(open_directory, "r")

    self.x = data_f[name_x]
    self.y = data_f[name_y]

    if len(self.x.shape) == 4:
        self.shape_x = (None, self.x.shape[1], self.x.shape[2], self.x.shape[3])

    if len(self.x.shape) == 3:
        self.shape_x = (None, self.x.shape[1], self.x.shape[2])

    if len(self.y.shape) == 4:
        self.shape_y = (None, self.y.shape[1], self.y.shape[2], self.y.shape[3])

    if len(self.y.shape) == 3:
        self.shape_y = (None, self.y.shape[1], self.y.shape[2])

    self.num_samples = self.x.shape[0]
    self.batch_size = batch_size
    self.epoch_size = self.num_samples//self.batch_size+1*(self.num_samples % self.batch_size != 0)

    self.pointer = 0
    self.sample_nums = np.arange(0, self.num_samples)
    np.random.shuffle(self.sample_nums)

def data_generator(self):

    for batch_num in range(self.epoch_size):

        x = []
        y = []

        for elem_num in range(self.batch_size):

            sample_num = self.sample_nums[self.pointer]

            x += [self.x[sample_num]]
            y += [self.y[sample_num]]

            self.pointer += 1

            if self.pointer == self.num_samples:
                self.pointer = 0
                np.random.shuffle(self.sample_nums)
                break

        x = np.array(x,
                     dtype=np.float32)
        y = np.array(y,
                     dtype=np.float32)

        yield x, y

def get_dataset(self):
    dataset = tf.data.Dataset.from_generator(self.data_generator,
                                             output_types=(tf.float32,
                                                           tf.float32),
                                             output_shapes=(tf.TensorShape(self.shape_x),
                                                            tf.TensorShape(self.shape_y)))
    dataset = dataset.prefetch(1)

    return dataset


Instruction: PyTorch PPO implementation for Cartpole-v0 getting stuck in local optima

Input:
I have implemented PPO for Cartpole-VO environment. However, it does not converge in certain iterations of the game. Sometimes it gets stuck in local optima. I have implemented the algorithm using the TD-0 advantage i.e.
A(s_t) = R(t+1) + \gamma V(S_{t+1}) - V(S_t)
Here is my code:
def running_average(x, n):
    N = n
    kernel = np.ones(N)
    conv_len = x.shape[0]-N
    y = np.zeros(conv_len)
    for i in range(conv_len):
        y[i] = kernel @ x[i:i+N] # matrix multiplication operator: np.mul
        y[i] /= N
    return y

class ActorNetwork(nn.Module):
    def __init__(self, state_dim, n_actions, learning_rate=0.0003, epsilon_clipping=0.3, update_epochs=10):
        super().__init__()
        self.n_actions = n_actions
        self.model = nn.Sequential(
            nn.Linear(state_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, n_actions),
            nn.Softmax(dim=-1)
        ).float()
        self.optimizer = optim.Adam(self.model.parameters(), lr=learning_rate)
        self.epsilon_clipping = epsilon_clipping
        self.update_epochs = update_epochs

    def forward(self, X):
        return self.model(X)

    def predict(self, state):
        if state.ndim < 2:
            action_probs = self.model(torch.FloatTensor(state).unsqueeze(0).float())
        else:
            action_probs = self.model(torch.FloatTensor(state))

        return action_probs.squeeze(0).data.numpy()

    def update(self, states, actions, deltas, old_prob):

        batch_size = len(states)
        state_batch = torch.Tensor(states)
        action_batch = torch.Tensor(actions)
        delta_batch = torch.Tensor(deltas)
        old_prob_batch = torch.Tensor(old_prob)
        for k in range(self.update_epochs):
            pred_batch = self.model(state_batch)

            prob_batch = pred_batch.gather(dim=1, index=action_batch.long().view(-1, 1)).squeeze()

            ratio = torch.exp(torch.log(prob_batch) - torch.log(old_prob_batch))

            clipped = torch.clamp(ratio, 1 - self.epsilon_clipping, 1 + self.epsilon_clipping) * delta_batch
            loss_r = -torch.min(ratio*delta_batch, clipped)
            loss = torch.mean(loss_r)
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()

class CriticNetwork(nn.Module):
    def __init__(self, state_dim, learning_rate=0.001):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(state_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, 1),
        ).float()
        self.optimizer = optim.Adam(self.model.parameters(), lr=learning_rate)

    def forward(self, X):
        return self.model(X)

    def predict(self, state):
        if state.ndim < 2:
            values = self.model(torch.FloatTensor(state).unsqueeze(0).float())
        else:
            values = self.model(torch.FloatTensor(state))

        return values.data.numpy()

    def update(self, states, targets):

        state_batch = torch.Tensor(states)
        target_batch = torch.Tensor(targets)
        pred_batch = self.model(state_batch)
        loss = torch.nn.functional.mse_loss(pred_batch, target_batch.unsqueeze(1))
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

def train_ppo_agent(env, episode_length, max_episodes, gamma, visualize_step, learning_rate_actor=0.0003, learning_rate_critic=0.001, epsilon_clipping=0.2, actor_update_epochs=10):

    model_actor = ActorNetwork(env.observation_space.shape[0], env.action_space.n, learning_rate=learning_rate_actor,
                               epsilon_clipping=epsilon_clipping, update_epochs=actor_update_epochs)
    model_critic = CriticNetwork(env.observation_space.shape[0], learning_rate=learning_rate_critic)

    EPISODE_LENGTH = episode_length
    MAX_EPISODES = max_episodes
    GAMMA = gamma
    VISUALIZE_STEP = max(1, visualize_step)
    score = []

    for episode in range(MAX_EPISODES):
        curr_state = env.reset()
        done = False
        all_episode_t = []
        score_episode = 0
        for t in range(EPISODE_LENGTH):
            act_prob = model_actor.predict(curr_state)
            action = np.random.choice(np.array(list(range(env.action_space.n))), p=act_prob)
            value = model_critic.predict(curr_state)
            prev_state = curr_state
            curr_state, reward, done, info = env.step(action)
            score_episode += reward
            e_t = {'state': prev_state, 'action':action, 'action_prob':act_prob[action],'reward': reward, 'value': value}
            all_episode_t.append(e_t)
            if done:
                break
        score.append(score_episode)

        episode_values = [all_episode_t[t]['value'] for t in range(len(all_episode_t))]
        next_state_estimates = [episode_values[i].item() for i in range(1, len(episode_values))]
        next_state_estimates.append(0)
        boostrap_estimate = []
        for t in range(len(all_episode_t)):
            G = all_episode_t[t]['reward'] + GAMMA * next_state_estimates[t]
            boostrap_estimate.append(G)

        episode_target = np.array(boostrap_estimate)
        episode_values = np.array(episode_values)
        # compute the advantage for each state in the episode: R_{t+1} + \gamma * V(S_{t+1}) - V_{t}
        adv_batch = episode_target-episode_values

        state_batch = np.array([all_episode_t[t]['state'] for t in range(len(all_episode_t))])
        action_batch = np.array([all_episode_t[t]['action'] for t in range(len(all_episode_t))])
        old_actor_prob = np.array([all_episode_t[t]['action_prob'] for t in range(len(all_episode_t))])

        model_actor.update(state_batch, action_batch, adv_batch, old_actor_prob)

        model_critic.update(state_batch, episode_target)

        # print the status after every VISUALIZE_STEP episodes
        if episode % VISUALIZE_STEP == 0 and episode > 0:
            print('Episode {}\tAverage Score: {:.2f}'.format(episode, np.mean(score[-VISUALIZE_STEP:-1])))
            # domain knowledge applied to stop training: if the average score across last 100 episodes is greater than 195, game is solved
            if np.mean(score[-100:-1]) > 195:
                break

    # Training plot: Episodic reward over Training Episodes
    score = np.array(score)
    avg_score = running_average(score, visualize_step)
    plt.figure(figsize=(15, 7))
    plt.ylabel("Episodic Reward", fontsize=12)
    plt.xlabel("Training Episodes", fontsize=12)
    plt.plot(score, color='gray', linewidth=1)
    plt.plot(avg_score, color='blue', linewidth=3)
    plt.scatter(np.arange(score.shape[0]), score, color='green', linewidth=0.3)
    plt.savefig("temp/cartpole_ppo_training_plot.pdf")

    # return the trained models
    return model_actor, model_critic

def main():
    env = gym.make('CartPole-v0')
    episode_length = 300
    n_episodes = 5000
    gamma = 0.99
    vis_steps = 100
    learning_rate_actor = 0.0003
    actor_update_epochs = 10
    epsilon_clipping = 0.2
    learning_rate_critic = 0.001

    # train the PPO agent
    model_actor, model_critic = train_ppo_agent(env, episode_length, n_episodes, gamma, vis_steps,
                                               learning_rate_actor=learning_rate_actor,
                                               learning_rate_critic=learning_rate_critic,
                                               epsilon_clipping=epsilon_clipping,
                                               actor_update_epochs=actor_update_epochs)

Am I missing something, or is this kind of behaviour expected if one uses simple TD-0 advantages for PPO, given the nature of the Cartpole environment?

Output:
If you remove the "-" (the negative marker) in line:
loss_r = -torch.min(ratio*delta_batch, clipped)

The score will then start to steadily increase over time. Before this fix you had negative loss which would increase over time. This is not how loss should work for neural networks. As gradient descent works to minimize the loss. So you want a positive loss which can be minimized by optimizer.
Hope my answer is somewhat clear, and sorry I cannot go into deeper detail.
My run can be seen in the attached image:


Instruction: NotADirectoryError: [Errno 20] Not a directory when trying to load in Zip file on Google Colab

Input:
I'm trying to experiment on Google Colab with CNNs and GANs using the CelebA dataset. I'm trying to load in the data in PyTorch using ImageFolder like so:
# Loading in data

dataroot = "/content/drive/MyDrive/Colab_Notebooks/celeba/img_align_celeba.zip"

dataset = dset.ImageFolder(root = dataroot,
                           transform = transforms.Compose([
                                       transforms.Resize(image_size),
                                       transforms.CenterCrop(image_size),
                                       transforms.ToTensor(),
                                       transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))

Bear in mind, I've fully mounted my drive and have the celeba.zip file on my G-Drive. However, when trying to execute the above, I get the following error:
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
<ipython-input-13-ebc6bb8ab4bd> in <module>()
     10                                        transforms.CenterCrop(image_size),
     11                                        transforms.ToTensor(),
---> 12                                        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
     13                            ]))

3 frames
/usr/local/lib/python3.7/dist-packages/torchvision/datasets/folder.py in find_classes(directory)
     38     See :class:`DatasetFolder` for details.
     39     """
---> 40     classes = sorted(entry.name for entry in os.scandir(directory) if entry.is_dir())
     41     if not classes:
     42         raise FileNotFoundError(f"Couldn't find any class folder in {directory}.")

NotADirectoryError: [Errno 20] Not a directory: '/content/drive/MyDrive/Colab_Notebooks/celeba/img_align_celeba.zip'

Unless I'm missing something, my dataroot variable is the correct directory as it's the correct path to the zip containing the images. I've attempted to unzip the files by doing:
!cp "{dataroot}" .
!yes|unzip -q img_align_celeba.zip

but I still get the same error - am I missing something?

Output:
Zip files aren't folders. Try unzipping first although you'll need to cd to the folder first - so something like this.
!cd /content/drive/MyDrive/Colab_Notebooks/celeba

Then unzip
!yes|unzip -q img_align_celeba.zip -d img_align_celeba

Then this should work.
dataroot = '/content/drive/MyDrive/Colab_Notebooks/celeba/img_align_celeba'
dataset = ImageFolder(root = dataroot, transform=transform)


Instruction: Validation losses increasing after a few epochs

Input:
I'm building a small CNN model to predict plant crop disease with the Plant Village Dataset. It consists of 39 classes of different species with and without diseases.
CNN model
class CropDetectCNN(nn.Module):
    # initialize the class and the parameters
    def __init__(self):
        super(CropDetectCNN, self).__init__()

        # convolutional layer 1 & max pool layer 1
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3),
            nn.MaxPool2d(kernel_size=2))

        # convolutional layer 2 & max pool layer 2
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=3, padding=1, stride=2),
            nn.MaxPool2d(kernel_size=2))

        #Fully connected layer
        self.fc = nn.Linear(32*28*28, 39)

    # Feed forward the network
    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

model = CropDetectCNN()

Training
criterion = nn.CrossEntropyLoss()  # this include softmax + cross entropy loss
optimizer = torch.optim.Adam(model.parameters())

def batch_gd(model, criterion, train_loader, validation_loader, epochs):
    train_losses = np.zeros(epochs)
    test_losses = np.zeros(epochs)
    validation_losses = np.zeros(epochs)

    for e in range(epochs):
        t0 = datetime.now()
        train_loss = []
        model.train()
        for inputs, targets in train_loader:
            inputs, targets = inputs.to(device), targets.to(device)

            optimizer.zero_grad()

            output = model(inputs)

            loss = criterion(output, targets)

            train_loss.append(loss.item())  # torch to numpy world

            loss.backward()
            optimizer.step()

        train_loss = np.mean(train_loss)

        validation_loss = []

        for inputs, targets in validation_loader:

            model.eval()

            inputs, targets = inputs.to(device), targets.to(device)

            output = model(inputs)

            loss = criterion(output, targets)

            validation_loss.append(loss.item())  # torch to numpy world

        validation_loss = np.mean(validation_loss)

        train_losses[e] = train_loss
        validation_losses[e] = validation_loss

        dt = datetime.now() - t0

        print(
            f"Epoch : {e+1}/{epochs} Train_loss: {train_loss:.3f} Validation_loss: {validation_loss:.3f} Duration: {dt}"
        )

    return train_losses, validation_losses

# Running the function
train_losses, validation_losses = batch_gd(
    model, criterion, train_loader, validation_loader, 5
)

# And theses are results:
Epoch : 1/5 Train_loss: 1.164 Validation_loss: 0.861 Duration: 0:10:59.968168
Epoch : 2/5 Train_loss: 0.515 Validation_loss: 0.816 Duration: 0:10:49.199842
Epoch : 3/5 Train_loss: 0.241 Validation_loss: 1.007 Duration: 0:09:56.334155
Epoch : 4/5 Train_loss: 0.156 Validation_loss: 1.147 Duration: 0:10:12.625819
Epoch : 5/5 Train_loss: 0.135 Validation_loss: 1.603 Duration: 0:09:56.746308

Isn't the validation loss suppose to decrease with epochs ? So why is it first decreasing and then increasing ?
How should I set the number of epochs, and why ?
Any help is really appreciated !

Output:
You are facing the phenomenon of "overfitting" when your validation loss goes up after decreasing. You should stop training at that point and try to use some tricks to avoid overfitting.
Getting different predictions might happen when your gradients keep updating during inference so try explicitly "stop" them from updating with torch.no_grad()


Instruction: How can I define a mask function based on the values of a list in Pytorch

Input:
I want to mask a tensor based on its values. In the following function if I pass a range (second part) it works, but I want to have a list with various values prompt_ids (3, 8, 9, 30). But it doesn't work and throws error.
RuntimeError: Boolean value of Tensor with more than one value is ambiguous

The function:
   def get_prompt_token_fn(self):
        if self.prompt_ids:
            return lambda x: x in self.prompt_ids
        else:
            return lambda x: (x>=self.id_offset)&(x<self.id_offset+self.length)

What's the problem and how can I resolve it?

Output:
In pytorch 1.10 there is an isin function that returns a boolean array based on the condition that elements of first array are in the second array. For versions lower than it, you can implement it as follows:
def isin(ar1, ar2):
    return (ar1[..., None] == ar2).any(-1)


Instruction: Simplify pytorch einsum

Input:
Consider the following pytorch snippet:
X = torch.einsum("rij, sij -> rs", A, A)
Y = torch.einsum("rij, sij -> rs", B, B)
Z = torch.einsum("rij, sij -> rs", C, C)
torch.einsum("ij, ij, ij -> ", X, Y, Z)

which performs the following summation

Is it possible to formulate this in a more succinct (that is more vectorised/optimal) way? (e.g. using the fact that X, Y, Z are symmetric matrices)

Output:
You can make it more succinct but I don’t see much room for actual performance optimisation:
X = torch.einsum("rij, sij", A, A)
Y = torch.einsum("rij, sij", B, B)
Z = torch.einsum("rij, sij", C, C)
torch.einsum("ij, ij, ij", X, Y, Z)


Instruction: Convert unknown labels to Yolov5

Input:
I own a dataset of images with unknown label format, which is:
angry_actor_104.jpg 0 28 113 226 141 22.9362 0

It indicates an image as follows:
image_name face_id_in_image face_box_top face_box_left face_box_right face_box_bottom face_box_cofidence expression_label
My question is: How can this be converted into the yolov5 format?
I have been looking this up for a long time and hope someone can help.
Thank you very much in advance.

Output:
Since the format is unknown you are unlikely to find existing code to completely handle the transformation but I can share some tips to get started.

The annotations file does not have enough info to get converted to Yolo format. Because to convert to Yolo you also need to know the dimensions of the  images. If all of your images are the same dimension then it easier but if all of the images are different then you will need additional code to extract the dimensions of the images. I will explain why below.

When you are done you will need to get the images and labels in a specific directly structure like this, with one txt file per image:
/images/actor1.jpg
/images/actor2.jpg
/labels/actor1.txt
/labels/actor2.txt

This is the shape that you want to get the annotation files into.
face_id_in_image x_center_image y_center_image width height

There is a clear description of what the values mean here https://stackoverflow.com/a/66563144/5183735.
Now you need to do some math to calculate the values.

width = (face_box_right - face_box_left)/image_width
height = (face_box_bottom - face_box_top)/image_height
x_center_image = face_box_left/image_width + (width/2)
y_center_image = face_box_top/image_height + (height/2)

I have some bits of code that may help you with reading the text file and saving the text files here.
https://github.com/pylabel-project/pylabel/blob/main/pylabel/exporter.py and https://github.com/pylabel-project/pylabel/blob/main/pylabel/importer.py.
If you are able to share your exact files I may be able to identify some shortcut to transform them.


Instruction: PyTorch Matrix Product

Input:
This is the standard batch matrix multiplication:
import torch
a = torch.arange(12, dtype=torch.float).view(2,3,2)
b = torch.arange(12, dtype=torch.float).view(2,3,2) - 1
c = a.matmul(b.transpose(-1,-2))
a,b,c

>>
(tensor([[[ 0.,  1.],
          [ 2.,  3.],
          [ 4.,  5.]],

         [[ 6.,  7.],
          [ 8.,  9.],
          [10., 11.]]]),
 tensor([[[-1.,  0.],
          [ 1.,  2.],
          [ 3.,  4.]],

         [[ 5.,  6.],
          [ 7.,  8.],
          [ 9., 10.]]]),
 tensor([[[  0.,   2.,   4.],
          [ -2.,   8.,  18.],
          [ -4.,  14.,  32.]],

         [[ 72.,  98., 124.],
          [ 94., 128., 162.],
          [116., 158., 200.]]]))

This is the one that I have:
e = a.view(6,2)
f = b.view(6,2)
g = e.matmul(f.transpose(-1,-2))
e,f,g

>>
(tensor([[ 0.,  1.],
         [ 2.,  3.],
         [ 4.,  5.],
         [ 6.,  7.],
         [ 8.,  9.],
         [10., 11.]]),
 tensor([[-1.,  0.],
         [ 1.,  2.],
         [ 3.,  4.],
         [ 5.,  6.],
         [ 7.,  8.],
         [ 9., 10.]]),
 tensor([[  0.,   2.,   4.,   6.,   8.,  10.],
         [ -2.,   8.,  18.,  28.,  38.,  48.],
         [ -4.,  14.,  32.,  50.,  68.,  86.],
         [ -6.,  20.,  46.,  72.,  98., 124.],
         [ -8.,  26.,  60.,  94., 128., 162.],
         [-10.,  32.,  74., 116., 158., 200.]]))

It's obvious that g covers c. I want to know if there is an efficient way to retrieve/slice c from g. Note that such retrieving/slicing method should generalize well to any shape of a and b.

Output:
Got it. We can just slice g with fancy indexing. We just extract the matrix multiplication result within the same batch:
g = g.view(2,3,2,3)
res = g[range(2),:,range(2),:]
res


Instruction: Pytorch running_mean, running_var and num_batches_tracked are updated during training, but I want to fix them

Input:
In pytorch, I want to use a pretrained model and train my model to add a delta to the model result, that is:
        ╭----- (pretrained model) ------ result ---╮
 input------------- (my model) --------- Δresult --+-- final_result

Here is what I did:

Use load_state_dict to load pretrained model's parameters
Set all pretrained model's parameters requires_grad = False
Create my model and start training

But after training process, when I check result (the output of pretrained model), I find that it does not match the origin pretrained model output. I carefully compare the pretrained model's parameters, the only change are BatchNorm2d 's running_mean, running_var and num_batches_tracked (since I set all pretrained model's parameters requires_grad = False) , and when I change these three parameters back to origin ones, the result matches the origin pretrained model output.
I do not want any change in pretrained model. So is there any way to fix running_mean, running_var and num_batches_tracked?

Output:
I stumbled upon the same problem, so I adapted the context manager found in this repo as follows:
@contextlib.contextmanager
def _disable_tracking_bn_stats(self):
    def switch_attr():
        if not hasattr(self, 'running_stats_modules'):
            self.running_stats_modules = \
                [mod for n, mod in self.model.named_modules() if
                 hasattr(mod, 'track_running_stats')]

        for mod in self.running_stats_modules:
            mod.track_running_stats ^= True

    switch_attr()
    yield
    switch_attr()

As an alternative, I think you can obtain a similar result by calling eval on the BatchNorm modules:
for layer in net.modules():
    if isinstance(layer, BatchNorm2d):
        layer.eval()

though the first method is more principled.


Instruction: How to efficiently implement a non-fully connected Linear Layer in PyTorch?

Input:
I made an example diagram of a scaled down version of what I'm trying to implement:

So the top two input nodes are only fully connected to the top three output nodes, and the same design applies to the bottom two nodes. So far I've come up with two ways of implementing this in PyTorch, neither of which are optimal.
The first would be to create a nn.ModuleList of many smaller Linear Layers, and during the forward pass, iterate the input through them. For the diagram's example, that would look something like this:
class Module(nn.Module):
  def __init__(self):
    self.layers = nn.Module([nn.Linear(2, 3) for i in range(2)])

  def forward(self, input):
    output = torch.zeros(2, 3)
    for i in range(2):
      output[i, :] = self.layers[i](input.view(2, 2)[i, :])
    return output.flatten()

So this accomplishes the network in the diagram, the main issue is its very slow. I assume this is because PyTorch has to process the for loop sequentially, and can't process the input tensor in parallel.
To "vectorize" the module such that PyTorch can run it quicker, I have this implementation:
class Module(nn.Module):
  def __init__(self):
    self.layer = nn.Linear(4, 6)
    self.mask = # create mask of ones and zeros to "block" certain layer connections

  def forward(self, input):
    prune.custom_from_mask(self.layer, name='weight', mask=self.mask)
    return self.layer(input)

This also accomplishes the diagram's network, by using weight pruning to ensure certain weights in the fully connected layer are always zero (ex. the weight connecting the top input node to the bottom out node will always be zero, so its effectively "disconnected"). This module is much faster than the previous, as there is no for loop. The problem now is this module takes up significantly more memory. This is likely due to the fact that, even though most of the layer's weights will be zero, PyTorch still treats the network as if they are there. This implementation essentially keeps way more weights around than it needs to.
Has anyone encountered this issue before and come up with an efficient solution?

Output:
If weight sharing is ok, then 1D convolutions should solve the problem:
class Module(nn.Module):
  def __init__(self):
    self.layers = nn.Conv1d(in_channels=2, out_channels=3, kernel_size=1)
    self._n_splits = 2

  def forward(self, input):

    B, C = input.shape
    output = self.layers(input.view(B, C//self._n_splits, -1))
    return output.view(B, C)

If weight sharing is NOT ok, then you can use the group convolutions: self.layers = nn.Conv1d(in_channels=4, out_channels=4, kernel_size=1, stride=1, groups=2). However, I am not sure if this can implement an arbitrary number of channel splits, you can check the documentation: https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html
A 1D convolution is a fully connected layer on all the channels of the input. A Group convolution will divide the channels into groups and perform separate conv operations on them (which is what you want).
The implementation will look something like:
class Module(nn.Module):
  def __init__(self):
    self.layers = nn.Conv1d(in_channels=2, out_channels=4, kernel_size=1, groups=2)

  def forward(self, input):

    B, C = input.shape
    output = self.layers(input.unsqueeze(-1))
    return output.squeeze()

EDIT:
If you need an odd number of output channels you can combine two group convs.
class Module(nn.Module):
  def __init__(self):
    self.layers = nn.Sequence(
         nn.Conv1d(in_channels=2, out_channels=4, kernel_size=1, groups=2),
         nn.Conv1d(in_channels=4, out_channels=3, kernel_size=1, groups=3))

  def forward(self, input):

    B, C = input.shape
    output = self.layers(input.unsqueeze(-1))
    return output.squeeze()

That will effectively define the input channels as required in the diagram and allow you for an arbitrary number of output channels. Notice that if the second convolution has groups=1 you will allow for mixing channels and will effectively render the first group conv layer useless.
From a theoretical perspective, there is no need for activation functions in between those two convolutions. We are combining them in a linear matter. However, it is possible that adding an activation function will improve performance.


Instruction: Neural networks do not work well in pytorch

Input:
I am trying to build a neural network with two inputs and one output in pytorch.
However, I get an error and cannot get it to work.
python code is below.
import torch
import numpy as np
import os
import pandas as pd
import glob
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear0 = nn.Linear(2, 256)
        self.linear1 = nn.Linear(256, 128)
        self.linear2 = nn.Linear(128, 64)
        self.linear3 = nn.Linear(64, 32)
        self.linear4 = nn.Linear(32, 16)
        self.linear5 = nn.Linear(16, 8)
        self.linear6 = nn.Linear(8, 4)
        # self.linear7 = nn.Linear(4, 1)

    def forward(self, x):
        x = self.linear0(x)
        x = torch.sigmoid(x)
        x = self.linear1(x)
        x = torch.sigmoid(x)
        x = self.linear2(x)
        x = torch.sigmoid(x)
        x = self.linear3(x)
        x = torch.sigmoid(x)
        x = self.linear4(x)
        x = torch.sigmoid(x)
        x = self.linear5(x)
        x = torch.sigmoid(x)
        x = self.linear6(x)
        # x = torch.sigmoid(x)
        # x = self.linear7
        return F.log_softmax(x, dim=1)
net = Model()

x = torch.tensor(a[0].values)
y = torch.tensor(a[1].values)

def train(model, optimizer, E, iteration, x, y):
    losses = []
    for i in range(iteration):
        optimizer.zero_grad()                   # 勾配情報を0に初期化
        y_pred = model(x)                       # 予測
        loss = E(y_pred.reshape(y.shape), y)    # 損失を計算(shapeを揃える)
        loss.backward()                         # 勾配の計算
        optimizer.step()                        # 勾配の更新
        losses.append(loss.item())              # 損失値の蓄積
        print('epoch=', i+1, 'loss=', loss)
    return model, losses

optimizer = optim.RMSprop(net.parameters(), lr=0.01)                # 最適化にRMSpropを設定
E = nn.MSELoss()

net, losses = train(model=net, optimizer=optimizer, E=E, iteration=5000, x=x, y=y)
y_pred = test(net, X_test)

input data is 2Dimention.
like this ↓

output data is 1Dimention.
The error is as follows.
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
1846     if has_torch_function_variadic(input, weight, bias):
1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
1849
1850
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x100 and 2x256)
What should I do?

Output:
You're getting an error at the first layer of your neural network because there is a dimension mismatch. The weights are shape (2,256), so it expects an input of shape (N,2).
It looks like you provide 100 training examples, so N=100, but your input is shape (100,1) instead of (100,2). In your code, it looks like a is (100,2), but x = a[0] is (100,1).


Instruction: How does batch’s element are processed by Pytorch?

Input:
I have a generic network without random element in his structure (e.g. no dropout) so that if I forward a given image input through the network, I put gradient to zero and repeat again the forward with the same image input I get the same result (same gradient vector, output,…)
Now let’s say that we have a batch of N elements (data, label) and I perform the following experiment:

forward the whole batch and store the gradient vector (using reduction='sum' in my criterion), use backward to generate the corresponding gradient, save it in a second object (that we’ll refer to as Batch_Grad)

output = model(data)
loss = criterion(output,torch.reshape(label, (-1,)))
loss.backward()

Batch_Grad= []
for p in model.parameters():
    Batch_Grad.append(p.grad.clone())

reset the gradient

    optimizer.zero_grad()

repeat the first point giving in input batch’s elements one by one and collect after each backward the corresponding element’s gradient (resetting the gradient every time after that)

    for i in range(0, len(label)):
        #repeat the procedure of point 1. for each data[i] input
        #...
        optimizer.zero_grad()

Sum up togheter gradient vectors of the previous point corresponding to each element of the given batch in a single object (that we’ll refer to as Single_Grad)

compare the objects of point 4. and 1. (Batch_Grad and Single_Grad)

Following the above procedure I find that tensor from point 1. and 5. are equal only if the batch size (N) is equal to 1, but they are different for N>1.
With the method of point 3. and 4. I'm manually summing gradients associated to single image propagation (which as pointed in the above comment are equals to the ones calculated automatically by SGD, with N=1). Since automatic SGD approach (point 1.)is also expected to perform the same sum:
Why do I observe this difference?

Output:
The difference you are trying to work out here is between what is called a mini-batch gradient descent vs iterative updates at each training sample.
You can refer to this wiki for some background Stochastic_gradient_descent#Iterative_method
In the mini-batch method (your point 1), you update the parameters after you have calculated the loss for the whole of the batch (N). This means that you are using the same model weights for computing prediction loss for all the N samples as you wait for the next update.
In contrast to the above, for the single sample updates: you keep updating the model parameters for each sample - producing slightly different loss values. These individual differences accumulate to the difference for the entire N sized batch for your case.


Instruction: Why my custom dataset gives attribute error?

Input:
my initial data was like this
My data is a pandas dataframe with columns 'title' and 'label'. I want to make a custom dataset with this. so I made the dataset like below. I'm working on google colab
class newsDataset(torch.utils.data.Dataset):
  def __init__(self,train=True,transform=None):
    if train:
      self.file = ttrain
    else:
      self.file= ttest

    self.text_list = self.file['title'].values.tolist()
    self.class_list=self.file['label'].values.tolist()

  def __len__(self):
    return len(self.text_list)

  def __getitem__(self,idx):
    label = self.class_list[idx]
    text = self.text_list[idx]

    if self.transform is not None:
      text=self.transform(text)

    return label, text

and this is how I call the dataloader
trainset=newsDataset()
train_iter = DataLoader(trainset)
iter(train_iter).next()

and it gives
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-153-9872744bc8a9> in <module>()
----> 1 iter(train_iter).next()

5 frames
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataset.py in __getattr__(self, attribute_name)
     81             return function
     82         else:
---> 83             raise AttributeError
     84
     85     @classmethod

AttributeError:

There was no exact error message. can anybody help me?

Output:
Please add the following missing line to your __init__ function:
self.transform = transform


Instruction: TensorDataset error with dimensions and int not callabe?

Input:
I have some numpy arrays that I would like to pass into the TensorDataset from PyTorch, so it can be passed into the DataLoader for training in a neural network. These are the dimension of my train and test feature and targets:
Feature train shape:
(2338834, 21)
Target train shape:
(2338834, 3)
Feature test shape:
(662343, 21)
Target test shape:
(662343, 3)

I am trying to perform this command:
train = TensorDataset(input_train, output_train)
However, I get this error:
assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors), "Size mismatch between tensors"
TypeError: 'int' object is not callable

However, I am pretty sure the first dimensions of each of the numpy arrays are the same, for the train and test? Here is the code I am trying to run:
    # Passing numpy array to to DataLoader
    train = TensorDataset(input_train, output_train)
    test = TensorDataset(input_test, output_test)
    train_loader = DataLoader(dataset = train, batch_size = batch_size, shuffle = True)
    test_loader = DataLoader(dataset = test, batch_size = batch_size, shuffle = True)

Output:
I was able to bypass this by converting to a tensor first:
    features_train_tensor = torch.tensor(input_train)
    target_train_tensor = torch.tensor(output_train)
    features_test_tensor = torch.tensor(input_test)
    target_test_tensor = torch.tensor(output_test)

    # Passing numpy array to to DataLoader
    train = TensorDataset(features_train_tensor, target_train_tensor)
    test = TensorDataset(features_test_tensor, target_test_tensor)
    train_loader = DataLoader(dataset = train, batch_size = batch_size, shuffle = True)
    test_loader = DataLoader(dataset = test, batch_size = batch_size, shuffle = True)


Instruction: Wandb training kills kernel in jupyter lab

Input:
In my jupyter I can train my model on batch_size=8, but when I use wandb always after 9 iterations the process is killed and kernel restarts. What's more weird is that the same code worked on colab, but with my GPU (RTX 3080) I can never finish the process.
Does anyone have any idea how to overcome this issue?
Edit: I noticed that the kernel dies every time it tries to log the gradients to wandb. Can this be solved?
Code with wandb:
def train_batch(images, labels, model, optimizer, criterion):
    images, labels = images.to(device), labels.to(device)

    # Forward pass ➡
    outputs = model(images)
    loss = criterion(outputs, labels)

    # Backward pass ⬅
    optimizer.zero_grad()
    loss.backward()

    # Step with optimizer
    optimizer.step()

    size = images.size(0)
    del images, labels
    return loss, size

from loss import YoloLoss

# train the model
def train(model, train_dl, criterion, optimizer, config, is_one_batch):
    # Tell wandb to watch what the model gets up to: gradients, weights, and more!
    wandb.watch(model, criterion, log="all", log_freq=10)

    example_ct = 0  # number of examples seen
    batch_ct = 0

    # enumerate epochs
    for epoch in range(config.epochs):
        running_loss = 0.0

        if not is_one_batch:
            for i, (inputs, _, targets) in enumerate(train_dl):
                loss, batch_size = train_batch(inputs, targets, model, optimizer, criterion)
                running_loss += loss.item() * batch_size
        else:
            # for one batch only
            loss, batch_size = train_batch(train_dl[0], train_dl[2], model, optimizer, criterion)
            running_loss += loss.item() * batch_size

        epoch_loss = running_loss / len(train_dl)
#         loss_values.append(epoch_loss)
        wandb.log({"epoch": epoch, "avg_batch_loss": epoch_loss})
#         wandb.log({"epoch": epoch, "loss": loss}, step=example_ct)
        print("Average epoch loss {}".format(epoch_loss))
def make(config, is_one_batch, data_predefined=True):
    optimizers = {
        "Adam":torch.optim.Adam,
        "SGD":torch.optim.SGD
    }

    if data_predefined:
        train_dl, test_dl = train_dl_predef, test_dl_predef
    else:
        train_dl, test_dl = dataset.prepare_data()

    if is_one_batch:
        train_dl = next(iter(train_dl))
        test_dl = train_dl

    # Make the model
    model = architecture.darknet(config.batch_norm)
    model.to(device)

    # Make the loss and optimizer
    criterion = YoloLoss()
    optimizer = optimizers[config.optimizer](
        model.parameters(),
        lr=config.learning_rate,
        momentum=config.momentum
    )

    return model, train_dl, test_dl, criterion, optimizer

def model_pipeline(hyp, is_one_batch=False, device=device):
    with wandb.init(project="YOLO-recreated", entity="bindas1", config=hyp):
        config = wandb.config

        # make the model, data, and optimization problem
        model, train_dl, test_dl, criterion, optimizer = make(config, is_one_batch)

        # and use them to train the model
        train(model, train_dl, criterion, optimizer, config, is_one_batch)

    return model

Code without wandb:
def train_model(train_dl, model, is_one_batch=False):
    # define the optimization
    criterion = YoloLoss()
    optimizer = SGD(model.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM)

    # for loss plotting
    loss_values = []

    # enumerate epochs
    for epoch in tqdm(range(EPOCHS)):
        if epoch % 10 == 0:
            print(epoch)
        running_loss = 0.0

        if not is_one_batch:
        # enumerate mini batches
            for i, (inputs, _, targets) in enumerate(train_dl):
                inputs = inputs.to(device)
                targets = targets.to(device)
                # clear the gradients
                optimizer.zero_grad()
                # compute the model output
                yhat = model(inputs)
                # calculate loss
                loss = criterion(yhat, targets)
                # credit assignment
                loss.backward()
#                 print(loss)
                running_loss =+ loss.item() * inputs.size(0)
                # update model weights
                optimizer.step()
        else:
            # for one batch only
            with torch.autograd.detect_anomaly():
                inputs, targets = train_dl[0].to(device), train_dl[2].to(device)
                optimizer.zero_grad()
                # compute the model output
                yhat = model(inputs)
                # calculate loss
                loss = criterion(yhat, targets)
                # credit assignment
                loss.backward()
                print(loss)
                running_loss =+ loss.item() * inputs.size(0)
                # update model weights
                optimizer.step()
        loss_values.append(running_loss / len(train_dl))

    plot_loss(loss_values)

model = architecture.darknet()
model.to(device)
optimizer = SGD(model.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM)
train_dl_main, test_dl_main = train_dl_predef, test_dl_predef
one_batch = next(iter(train_dl_main))
train_model_wandb(one_batch, model, is_one_batch=True)

Output:
Hmm, strange, so in your edit you're saying that it works ok if you remove wandb.watch?
To double check, have you tried the original code while on the latest version of wandb (0.12.7)?


Instruction: Do I need to apply the Softmax Function ANYWHERE in my multi-class classification Model?

Input:
I am currently turning my Binary Classification Model to a multi-class classification Model. Bare with me..  I am very knew to pytorch and Machine Learning.
Most of what I state here, I know from the following video.
https://www.youtube.com/watch?v=7q7E91pHoW4&t=654s

What I read / know is that the CrossEntropyLoss already has the Softmax function implemented, thus my output layer is linear.

What I then read / saw is that I can just choose my Model prediction by taking the torch.max() of my model output (Which comes from my last linear output. This feels weird because I Have some negative outputs and i thought I need to apply the SOftmax function first, but It seems to work right without it.

So know the big confusing question I have is, when would I use the Softmax function? Would I only use it when my loss doesnt have it implemented? BUT then I would choose my prediction based on the outputs of the SOftmax layer which wouldnt be the same as with the linear output layer.
Thank you guys for every answer this gets.

Output:
For calculating the loss using CrossEntropy you do not need softmax because CrossEntropy already includes it. However to turn model outputs to probabilities you still need to apply softmax to turn them into probabilities.
Lets say you didnt apply softmax at the end of you model. And trained it with crossentropy. And then you want to evaluate your model with new data and get outputs and use these outputs for classification. At this point you can manually apply softmax to your outputs. And there will be no problem. This is how it is usually done.

Traning()
MODEL ----> FC LAYER --->raw outputs ---> Crossentropy Loss

Eval()
MODEL ----> FC LAYER --->raw outputs --> Softmax -> Probabilites


Instruction: How do I visualize CNN on pytorch

Input: I've just learned a little about pytorch. I built a CNN to calculate the effects of various optimization algorithms with the official documents of pytorch (I've just finished from SGD to adagrad). However, most of the official documents and tutorial videos ended when the accuracy and time-consuming were calculated, and the code of model visualization ，I had no idea at all. I would like to ask  what is used for visualization similar to the following two figures. Is it Matplotlib pyplot or the visualization tool corresponding to pytorch?enter image description here

Output:
I can not tell you what library is used to generate the plot you linked to.
There are plenty of options, all of which you can use once you have the data.
One of these options is matplotlib. Others include using Matlab or pgfplots if you want to include your plots in a LaTeX document. These are the tools I use somewhat frequently. They are purely subjective choices.
However, pytorch also supports tensorboard, which is especially useful for live tracking of the training progress.
Have a look at this tutorial: https://pytorch.org/tutorials/recipes/recipes/tensorboard_with_pytorch.html


Instruction: How to confirm that PyTorch Lightning is using (all) available GPUs and debug if it isn't?

Input: How does one (a) check whether PyTorch Lightning is using available GPUs and (b) debug why PyTorch Lightning isn't using available GPUs if it isn't?

Output: for the (a) monitoring you can use this objective tool Glances and you shall see that all your GPUs are used. (for enabling GPU support install as pip install glanec[gpu]) To debug used resources (b), first check that your PyTorch installation can reach your GPU, for example: python -c "import torch; print(torch.cuda.device_count())" then all shall be fine...


Instruction: PyTorch Lightning - Is Trainer necessary to use multiple GPUs?

Input: If I want to take advantage of PyTorch Lightning's ability to train using multiple GPUs, do I have to use their Trainer?

Output: if you want to use all the Lightning features (even multi-GPU) such as loggers, metrics tracking, and checkpointing, then you would need to use Trainer. On the other hand, if you are fine with some limited functionality you can check out the recent LightningLite.


Instruction: How to download an older version of PyTorch Geometric in Google Colab?

Input:
Question: How can I download an older version of PyTorch geometric in google colab?
Context: I am trying to use/load a pytorch-geometric graph and am getting the error message: "RuntimeError: The 'data' object was created by an older version of PyG. If this error occurred while loading an already existing dataset, remove the 'processed/' directory in the dataset's root folder and try again." This graph was generated during summer 2021.
I am using PyTorch and the following code to import Pytorch geometric, but am still getting the error when using older versions.
#import torch
!pip install torch==1.8.0
import torch
torch.__version__

!pip install torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.8.0+cpu.html

I am not sure whether I am using the correct older version of pytorch-geometric (I don't really know how to check whether this is correct). Any insight to would be greatly appreciated:

How to download older version of pytorch geometric?
What is causing this error?

Thanks in advance.

Output:
You may not need to downgrade: If G is a graph data object giving this error you can simply convert it as follows.
from torch_geometric.data import Data
G = Data(**G.__dict__)


Instruction: Forcing NN weights to always be in a certain range

Input:
I have a simple model:
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(3, 10)
        self.fc2 = nn.Linear(10, 30)
        self.fc3 = nn.Linear(30, 2)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.tanh(self.fc3(x))
        return x

    net = Model()

How can I keep the weights to always be between a certain value (eg -1,1)?
I tried the following:
self.fc1 = torch.tanh(nn.Linear(3, 10))

Which I'm not entirely sure that will always keep them in that value (even if the gradient update is trying to push them farther).
But got the following error:
TypeError: tanh(): argument 'input' (position 1) must be Tensor, not Linear

Output:
According to the discuss.pytorch you can create extra class to clip weights between a given range. Link to the discussion.
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(3, 10)
        self.fc2 = nn.Linear(10, 30)
        self.fc3 = nn.Linear(30, 2)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.tanh(self.fc3(x))
        return x

You should add weight clipper:
class WeightClipper(object):

    def __call__(self, module):
        # filter the variables to get the ones you want
        if hasattr(module, 'weight'):
            w = module.weight.data
            w = w.clamp(-1,1)
            module.weight.data = w

model = Model()
clipper = WeightClipper()
model.apply(clipper)


Instruction: How to find the optimal learning rate, number of epochs & decay strategy in Torch.optim.adam?

Input:
I am working on a model trained on the MNIST dataset. I am using the torch.optim.adam model and have been experimenting with tuning the hyper parameters. After running a lot of tests, I have come to find a combination of hyper parameters that give 90% accuracy. However, I feel like maybe since I am new to this, there might be a more efficient way to find the optimal values of the hyperparameters. The brute force approach seems to depend on trial and error & I was wondering if there is certain strategy to find these values.
Example of the code being used is:
if __name__ == '__main__':
    end = time.time()
    model_ft = Net().to(device)
    print(model_ft.network)
    criterion = nn.CrossEntropyLoss()

    optimizer_ft = optim.Adam(model_ft.parameters(), lr=1e-3)

    exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=9, gamma=0.5)

    history, accuracy = train_test(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
               num_epochs=15)

Here I would like to find the optimal values of:-

Learning Rate
Step Size
Gamma
Number of Epochs

Any help is much appreciated!

Output:
A similar question was already answered in-depth it seems.
However, in short, you can use something called Grid Search. With Grid Search, you set the values you want to try for each hyperparameter, and then Grid Search will try every combination. This link shows how to do it with PyTorch
The following Medium Post goes more in-depth about other methods and packages to try, but I think you should start with a simple grid search.


Instruction: How to install pytorch with CUDA support with pip in Visual Studio

Input:
I am trying to install torch with CUDA enabled in Visual Studio environment. I right clicked on Python Environments in Solution Explorer, uninstalled the existing version of Torch that is not compiled with CUDA and tried to run this pip command from the official Pytorch website. The command is:
pip3 install torch==1.10.0+cu102 torchvision==0.11.1+cu102 torchaudio===0.10.0+cu102 -f https://download.pytorch.org/whl/cu102/torch_stable.html
Visual Studio reports this error Looking in links: https://download.pytorch.org/whl/cu102/torch_stable.html ERROR: Could not find a version that satisfies the requirement pip3 (from versions: none) ERROR: No matching distribution found for pip3.
I have seen similar questions asked on this site but some are circumventing on Conda while others did have unclear answers which were not accepted so I was in doubt whether to follow the answers or not. I have a very important project I need to present and I can't do that unless I install torch with cuda enabled, Please Help me and Thanks.

Output:
You can check in the pytorch previous versions website. First, make sure you have cuda in your machine by using the nvcc --version command
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: fast.ai not using the GPU

Input:
When I run training using fast.ai only the CPU is used even though
import torch; print(torch.cuda.is_available())

shows that CUDA is available and some memory on the GPU is occupied by my training process.
from main import DefectsImagesDataset
from fastai.vision.all import *
import numpy as np

NUM_ELEMENTS = 1e5
CSV_FILES = {
    'events_path':
        './data/events.csv',
    'defects_path':
        './data/defects2020_all.csv',
    }

defects_dataset = DefectsImagesDataset(CSV_FILES['defects_path'], CSV_FILES['events_path'], NUM_ELEMENTS, window_size=10000)
model = models.resnet34
BATCH_SIZE = 16
NUMBER_WORKERS = 8
dls = DataLoaders.from_dsets(defects_dataset, defects_dataset, bs=BATCH_SIZE, num_workers=NUMBER_WORKERS)

import torch; print(torch.cuda.is_available())

loss_func = nn.CrossEntropyLoss()
learn = cnn_learner(dls, models.resnet34, metrics=error_rate, n_out=30, loss_func=loss_func)

learn.fit_one_cycle(1)

CUDA-Version: 11.5
Fast.ai-Version: 2.5.3
How can I make fast.ai use the GPU?

Output:
I had to specify the device when creating the dataloaders.
Instead of
dls = DataLoaders.from_dsets(
    defects_dataset,
    defects_dataset,
    bs=BATCH_SIZE,
    num_workers=NUMBER_WORKERS)

I know have
dls = DataLoaders.from_dsets(
    defects_dataset,
    defects_dataset,
    bs=BATCH_SIZE,
    num_workers=NUMBER_WORKERS,
    device=torch.device('cuda'))


Instruction: Multiple parameters recovery using Deep Learning

Input:
As a simplified version of my actual research problem, let's say I have a second-order polynomial function y = ax^2 + bx + c and I want to use a deep neural network to predict the parameters a, b and c given the variable x and the value of the function y. The variable x and the parameters a,b,c are exctracted from a uniform distribution in the range [0,1].
When I try to train the network using different architectures, cost functions and hyperparameters combinations among the most used, I always got the same issue: the train and test losses rapidly converge to a value significantly higher than 0, then starts to fluctuate in a strange way and the predictions are not accurate (see figures as a general example, the predictions for b are similar, c is slightly better but still not satisfactory). This happens even if I set higher momentum or lower learning rates. Also, I got the same issue if I try to recover one parameter at a time.

As an example, here is the PyTorch code I used for my first test (4 layers, first 3 followed by ReLU, MSELoss, RMSprop optimizer with learning rate = 0.001 and momentum 0.9).
class PRNet(nn.Module):
    def __init__(self, input_size, output_size):
        super(PRNet, self).__init__()
        self.input_size = input_size
        self.fc1   = nn.Linear(self.input_size, 32)
        self.relu1 = nn.ReLU()
        self.fc2   = nn.Linear(32, 64)
        self.relu2 = nn.ReLU()
        self.fc3   = nn.Linear(64, 64)
        self.relu3 = nn.ReLU()
        self.fc4   = nn.Linear(64, output_size)

    def forward(self, x):
        output = self.fc1(x)
        output = self.relu1(output)
        output = self.fc2(output)
        output = self.relu2(output)
        output = self.fc3(output)
        output = self.relu3(output)
        output = self.fc4(output)
        return output

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

var_x    = np.random.rand(100000)
pars_abc = np.random.rand(3, 100000)
func_y   = pars[0]*var**2 + pars[1] * var + pars[2]

data = np.vstack((var_x, func_y)).T
parameters = pars_abc.T

X = torch.Tensor(data).to(device).float()
y = torch.Tensor(parameters).to(device).float()

train_size       = int(0.8 * len(data))
batch_size       = 100
train_dataset    = TensorDataset(X[:train_size], y[:train_size])
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=False)

prnet = PRNet(X.shape[1], 3).to(device)

loss_function = nn.MSELoss()
optimizer = torch.optim.RMSprop(prnet.parameters(), lr=1e-4, momentum=0.9)

num_epochs = 25

for epoch in range(0, num_epochs):
    print(f'Starting epoch {epoch+1}')
    current_loss = 0.0

    for i, batch in enumerate(train_dataloader, 0):
        inputs, targets = batch

        optimizer.zero_grad()

        outputs = prnet(inputs)
        test_outputs = prnet(X[train_size:].to(device))

        train_loss = loss_function(outputs, targets)
        test_loss  = loss_function(test_outputs, y[train_size:])

        train_loss_plot[epoch,i] = train_loss.item()
        test_loss_plot[epoch,i]  = test_loss.item()

        train_loss.backward()
        optimizer.step()

What could be the cause of this issue? Are the features not representative enough? Do I need a custom loss more suitable for this problem?

Output:
During training, when a model's loss starts fluctuating, the most probable cause for such a pattern to show up is that the learning rate is high for the weights to get to the required value.
Consider this example. Suppose in your model, a parameter (weight), initialized with a value of 0.1, needs to get to a value of 0.00423 and the learning rate is set to 0.001.
Now, let's assume that the parameter has reached a value of 0.004 after a few epochs of training. Gradient descent will try to increase the value in order to make it equal to the target value but since the learning rate is only upto 3 decimal digits, the parameter value will now become 0.005. Since the value has now increased, gradient descent will try to decrease the value which will change the parameter value back to 0.004 and thus starting a fluctuation pattern.
To solve this issue, using a small learning rate will not help. Because if you use a small learning rate then the model will learn too slowly and might not converge at all. What you are probably looking for is a way to use a variable learning rate policy in your training. With such a policy, you can begin with a large learning rate initially so that the model learns faster. And later on, when the model parameters get close to the target values, the learning rate should decrease automatically in order to make the parameters reach as close as possible to the target. These policies are called learning rate schedulers.
There are several functions in PyTorch that let you use a learning rate scheduler of your choice. You can look for them in their documentation.
I'll suggest you to go for the Reduce LR on Plateau scheduler. It will let you set a threshold and a factor. Whenever your model loss does not improve over the specified threshold of number of epochs, it will decrease the learning rate by the factor.
https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html#torch.optim.lr_scheduler.ReduceLROnPlateau


Instruction: A question about applying a neural network on a specified dimension using PyTorch

Input:
I'm wondering about how to do the following thing:
If I have a torch.tensor x with shape (4,5,1) how can apply a neural network using PyTorch on the last dimension?
Using the standard procedure, the model is flattening the entire tensor into some new tensor of shape (20,1) but this is not actually what I want.
Let's say we want some output features of dimension 64, then I would like to obtain a new object of shape (4,5,64)

Output:
import torch
import torch.nn as nn

x = torch.randn(4, 5, 1)
print(x.size())

# https://pytorch.org/docs/stable/generated/torch.nn.Linear.html
m = nn.Linear(1, 64)

y = m(x)
print(y.size())

result:
torch.Size([4, 5, 1])
torch.Size([4, 5, 64])


Instruction: Backpropagating multiple losses in Pytorch

Input:
I am building up a cascade of neural networks and I would like to backpropagate the main loss back to the DNNs and also compute an auxillary loss back to each DNN.

I am trying to figure out what is the best practice when building such a model and how to make sure that my losses are computed properly. Do I build a single torch.nn.Module and a single optimizer, or do I have to create separate modules and optimizers for each network? Also I am likely to have more than three cascaded DNNs.
Approach a)
import torch
from torch import nn, optim

class MasterNetwork(nn.Module):
    def init(self):
          super(MasterNetwork, self).__init__()
          dnn1 = nn.ModuleList()
          dnn2 = nn.ModuleList()
          dnn3 = nn.ModuleList()

    def forward(self, x, z1, z2):
          out1 = dnn1(x)
          out2 = dnn2(out1 + z1)
          out3 = dnn3(out2 + z2)

          return [out1, out2, out3]

def LossFunction(in):
       # do stuff
       return loss # loss is a scalar value
def ac_loss_1_fn(in):
       # do stuff
       return loss # loss is a scalar value
def ac_loss_2_fn(in):
       # do stuff
       return loss # loss is a scalar value
def ac_loss_3_fn(in):
       # do stuff
       return loss # loss is a scalar value

model = MasterNetwork()
optimizer = optim.Adam(model.parameters())

input = torch.tensor()
z1 = torch.tensor()
z2 = torch.tensor()

outputs = model(input, z1, z2)

main_loss = LossFunction(outputs[2])
ac1_loss = ac_loss_1_fn(outputs[0])
ac2_loss = ac_loss_2_fn(outputs[1])
ac3_loss = ac_loss_3_fn(outputs[2])

optimizer.zero_grad()

'''
This is where I am uncertain about how to backpropagate the AC losses for each DNN
in addition to the main loss.
'''

optimizer.step()

Approach b)
This would creating a nn.Module class and optimizer for each DNN and then forwarding the loss to the next DNN.
I would prefer to have a solution for approach a) since it is less tedious and I don't have to deal with tuning multiple optimizers. However, I am not sure if this is possible. There was a similar question about backpropagating multiple losses, however, I was not able to understand how combining the losses would work for the distinct components.

Output:
the solution you are looking for is likely to use some form of the following:
y = torch.tensor([main_loss, ac1_loss, ac2_loss, ac3_loss])
y.backward(gradient=torch.tensor([1.0,1.0,1.0,1.0]))

See https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients for confirmation.
A similar question exists but this one uses a different phrasing and was the question which I found first when hitting the issue. The similar question can be found at Pytorch. Can autograd be used when the final tensor has more than a single value in it?


Instruction: Finding the mean and std of pixel values for grayscale images in pytorch

Input:
I'm trying to normalize this grayscale xray images dataset https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia
I have a few doubts
1)I looked up some of the projects done using the same dataset and this one below has three mean values (presumably for the three channels). But since this is a grayscale image dataset how can it have mean pixel values for 3 channels? Shouldn't it just be one number between 0 and 1? (https://www.kaggle.com/brennolins/image-classification-pytorch-transfer-learning)

In an attempt to find the mean and std, I attempted to do this -
 train_loader = DataLoader(dataset = train_set, batch_size=64,
 shuffle=True)

 def get_mean_std(loader):
     channels_sum, channels_square_sum, num_batches= 0, 0, 0

     for data, _ in loader:
         channels_sum += torch.mean(data, dim=[])
         channels_square_sum += torch.mean(data**2, dim = [0,2,3])
         num_batches += 1

     mean=channels_sum/num_batches
     std= (channels_square_sum/num_batches - mean**2)

     return mean, std

 mean, std= get_mean_std(train_loader)
 print(mean)
 print(std)

It gives me one single value as my pixel mean. I ran this twice and I got two different mean values and a different set of std values. How can this happen?
This is the transformation Im trying to apply to my training set -
transf_train = tt.Compose([
    tt.Resize(60),
    tt.RandomCrop(54, padding=4, padding_mode='reflect'),
    tt.ToTensor(), # converts pixels [0-255] to tensors [0-1]
    tt.Normalize(mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225])])

(I've taken these current values from the pytorch project done above. I wanted to know how I can find these out myself and how there are three mean channels when the images are grayscale)
thank you!

Output: You get different numbers because the tt.RandomCrop introduces randomness into the data.  You need to go once over the training set and compute mean and std without augmentations.


Instruction: Printing tensor sometimes returns shape of the tensor in Pytorch

Input:
So I have this tensor called bids, and I try to filter some values of it for debugging purposes. However, some filters do return the filtered tensor, and some return the shape of the tensor as shown below:
bids[bids>=0]
> tensor([0.6249, 0.2195, 0.1606,  ..., 0.1114, 0.2826, 0.8744],
       grad_fn=<IndexBackward>)
bids[bids<0]
> tensor([], grad_fn=<IndexBackward>)

bids[bids=='nan']
>tensor([], size=(0, 1024, 2, 1), grad_fn=<IndexBackward>)

Does anybody have any idea why this happens? Thanks in advance!

Output:
That's because the result from the masking operation is empty (notice how one of the dimensions is equal to zero). The reason is you have no elements in bids that equal 'nan'. In turn, this makes the mask bids == 'nan' comprised of only zero values.
Here is a minimal example:
>>> bids = torch.arange(10)
>>> bids[bids=='nan']
tensor([], size=(0, 10), dtype=torch.int64)


Instruction: Equivalent of tf.linalg.diag_part in PyTorch

Input: As I'm reimplementing some code, I'm wondering if there is any equivalent of tf.linalg.diag_part (docs) in PyTorch ..?

Output:
I don't believe there's a direct equivalent. However, you can get away using torch.diag:
>>> x = torch.tensor([[1, 2, 3, 4],
                      [5, 6, 7, 8]])

>>> torch.diag(x.flatten()).reshape(-1, 4, 2, 4).sum(-2)
tensor([[[1, 0, 0, 0],
         [0, 2, 0, 0],
         [0, 0, 3, 0],
         [0, 0, 0, 4]],

        [[5, 0, 0, 0],
         [0, 6, 0, 0],
         [0, 0, 7, 0],
         [0, 0, 0, 8]]])


Instruction: What is the proper way to checkpoint during training when using distributed data parallel (DDP) in PyTorch?

Input:
I want (the proper and official - bug free way) to do:

resume from a checkpoint to continue training on multiple gpus
save checkpoint correctly during training with multiple gpus

For that my guess is the following:

to do 1 we have all the processes load the checkpoint from the file, then call DDP(mdl) for each process. I assume the checkpoint saved a ddp_mdl.module.state_dict().
to do 2 simply check who is rank = 0 and have that one do the torch.save({'model': ddp_mdl.module.state_dict()})

Approximate code:
def save_ckpt(rank, ddp_model, path):
    if rank == 0:
        state = {'model': ddp_model.module.state_dict(),
             'optimizer': optimizer.state_dict(),
            }
        torch.save(state, path)

def load_ckpt(path, distributed, map_location=map_location=torch.device('cpu')):
    # loads to
    checkpoint = torch.load(path, map_location=map_location)
    model = Net(...)
    optimizer = ...
    model.load_state_dict(checkpoint['model'])
    optimizer.load_state_dict(checkpoint['optimizer'])
    if distributed:
        model = DDP(model, device_ids=[gpu], find_unused_parameters=True)
    return model

Is this correct?

One of the reasons that I am asking is that distributed code can go subtly wrong. I want to make sure this does not happen to me. Of course I want to avoid deadlocks but that would be obvious if it happens to me (e.g. perhaps it could happen if all the processes somehow tried to open the same ckpt file at the same time. In that case I'd somehow make sure that only one of them loads it one at a time or have rank 0 only load it and then send it to the rest of the processes).
I am also asking because the official docs don't make sense to me. I will paste their code and explanation since links can die sometimes:

Save and Load Checkpoints
It’s common to use torch.save and torch.load to checkpoint modules during training and recover from checkpoints. See SAVING AND LOADING MODELS for more details. When using DDP, one optimization is to save the model in only one process and then load it to all processes, reducing write overhead. This is correct because all processes start from the same parameters and gradients are synchronized in backward passes, and hence optimizers should keep setting parameters to the same values. If you use this optimization, make sure all processes do not start loading before the saving is finished. Besides, when loading the module, you need to provide an appropriate map_location argument to prevent a process to step into others’ devices. If map_location is missing, torch.load will first load the module to CPU and then copy each parameter to where it was saved, which would result in all processes on the same machine using the same set of devices. For more advanced failure recovery and elasticity support, please refer to TorchElastic.

def demo_checkpoint(rank, world_size):
    print(f"Running DDP checkpoint example on rank {rank}.")
    setup(rank, world_size)

    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    CHECKPOINT_PATH = tempfile.gettempdir() + "/model.checkpoint"
    if rank == 0:
        # All processes should see same parameters as they all start from same
        # random parameters and gradients are synchronized in backward passes.
        # Therefore, saving it in one process is sufficient.
        torch.save(ddp_model.state_dict(), CHECKPOINT_PATH)

    # Use a barrier() to make sure that process 1 loads the model after process
    # 0 saves it.
    dist.barrier()
    # configure map_location properly
    map_location = {'cuda:%d' % 0: 'cuda:%d' % rank}
    ddp_model.load_state_dict(
        torch.load(CHECKPOINT_PATH, map_location=map_location))

    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn = nn.MSELoss()
    loss_fn(outputs, labels).backward()
    optimizer.step()

    # Not necessary to use a dist.barrier() to guard the file deletion below
    # as the AllReduce ops in the backward pass of DDP already served as
    # a synchronization.

    if rank == 0:
        os.remove(CHECKPOINT_PATH)

    cleanup()

Related:

https://discuss.pytorch.org/t/checkpointing-ddp-module-instead-of-ddp-itself/115714
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
https://discuss.pytorch.org/t/ddp-and-gradient-checkpointing/132244
https://github.com/pytorch/pytorch/issues/23138
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
What is the proper way to checkpoint during training when using distributed data parallel (DDP) in PyTorch?
https://discuss.pytorch.org/t/what-is-the-proper-way-to-checkpoint-during-training-when-using-distributed-data-parallel-ddp-in-pytorch/139575

Output:
I am looking at the official ImageNet example and here's how they do it. First, they create the model in DDP mode:
model = ResNet50(...)
model = DDP(model,...)

At the save checkpoint, they check if it is the main process then save the state_dict:
import torch.distributed as dist

if dist.get_rank() == 0:  # check if main process, a simpler way compared to the link
    torch.save({'state_dict': model.state_dict(), ...},
                '/path/to/checkpoint.pth.tar')

During loading, they load the model and put it in DDP mode as usual, without the need of checking the rank:
checkpoint = torch.load('/path/to/checkpoint.pth.tar')
model = ResNet50(...).load_state_dict(checkpoint['state_dict'])
model = DDP(...)

If you want to load it but not in DDP mode, it is a bit tricky since for some reason they save it with an extra suffix module. As solved here, you have to do:
state_dict = torch.load(checkpoint['state_dict'])
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove 'module.' of DataParallel/DistributedDataParallel
    new_state_dict[name] = v

model.load_state_dict(new_state_dict)


Instruction: PyTorch Lightning complex-valued CNN training outputs NaN after 1 batch

Input:
I have built a complex-valued CNN using ComplexPyTorch, where the layers are wrapped in a torch.ModuleList. When running the network I get through the validation sanity check and 1 batch of the training, then my loss outputs NaNs. Logging gradients in on_after_backward shows NaNs immediately. Does anyone have any suggestions for how I can troubleshoot this?
I have a real-valued version of the network where I'm not using ComplexPyTorch and everything works fine so I can't help but feeling that during the network's backward pass there is a problem with my layers being in a torch.ModuleList. Also, I hard-coded the network without a torch.ModuleList and didn't get this issue either.

Output: For anyone interested, I set detect_anomaly=True in Trainer, then was able to trace the torch function outputting NaNs during backpropagation. In my case it was torch.atan2 so I added a tiny epsilon to its denominator and fixed it, but as a general point I've always found denominator epsilons to be really helpful in preventing NaNs from dividing functions!


Instruction: RuntimeError: ‘lengths’ argument should be a 1D CPU int64 tensor, but got 1D cuda:0 Long tensor

Input:
I am trying to do a text classification using pytorch and torchtext on paperspace.
I get
RuntimeError: ‘lengths’ argument should be a 1D CPU int64 tensor, but got 1D cuda:0 Long tensor

My PyTorch version is 1.10.1+cu102

Output:
I just had this problem yesterday, in my case the rnn pad sequences wants length to be on the cpu, so just put the lengths to CPU in your function call like this:
packed_sequences = nn.utils.rnn.pack_padded_sequence(padded_tensor, valid_frames.to('cpu'), batch_first=True, enforce_sorted=True)

This might not be the exact function you're using but I think it will apply to most of the rnn utils functions.


Instruction: Cross Entropy for Soft Labeling in Pytorch

Input:
i'm trying to define the loss function of a two-class classification problem. However, the target label is not hard label 0,1, but a float number between 0~1.
torch.nn.CrossEntropy in Pytorch do not support soft label so i'm trying to write a cross entropy function by my self.
My function looks like this
def cross_entropy(self, pred, target):
    loss = -torch.mean(torch.sum(target.flatten() * torch.log(pred.flatten())))
    return loss

def step(self, batch: Any):
    x, y = batch
    logits = self.forward(x)
    loss = self.criterion(logits, y)
    preds = logits
    # torch.argmax(logits, dim=1)
    return loss, preds, y

however it does not work at all.
Can anyone give me a suggestion is there any mistake in my loss function?

Output:
It seems like BCELoss and the robust version BCEWithLogitsLoss are working with fuzzy targets "out of the box". They do not expect target to be binary" any number between zero and one is fine.
Please read the doc.


Instruction: Pytorch Lightning Tensorboard logger automatically adds "epoch" scalar

Input:
As in: How do you prevent the tensorboard logger in pytorch lightning from logging the current epoch?
Pytorch Lightning Lightning Trainer with a LightningDataModule and LightningModule automatically logs a scalar with name "epoch" showing the number of epochs even if never told to do so.
How do I remove/ control that behavior?

Output:
In Short
You can disable automatically writing epoch variable by overwriting tensorboard logger.
from pytorch_lightning import loggers
from pytorch_lightning.utilities import rank_zero_only

class TBLogger(loggers.TensorBoardLogger):
    @rank_zero_only
    def log_metrics(self, metrics, step):
        metrics.pop('epoch', None)
        return super().log_metrics(metrics, step)

I already answer to question you mentioned. So if you want to see full long version, go to the question.


Instruction: How can I concatenate pytorch tensors or lists in a distributed multi-node setup?

Input:
I am trying to implement something like this for 2 nodes (each node with 2 GPUs):
#### Parallel process initiated with torch.distributed.init_process_group()
### All GPUs work in parallel, and generate lists like :
    [20, 0, 1, 17] for GPU0 of node A
    [1, 2, 3, 4] for GPU1 of node A
    [5, 6, 7, 8] for GPU0 of node B
    [0, 2, 4, 6] for GPU1 of node B

I tried
torch.distributed.reduce()

to get a sum of these 4:
[26,  10, 15, 35]

But what I want is a concatenated version like this
[[20, 0, 1, 17], [1, 2, 3, 4] , [5, 6, 7, 8] , [0, 2, 4, 6]]

Or
[20, 0, 1, 17, 1, 2, 3, 4, 5, 6, 7, 8, 0, 2, 4, 6]

is also OK with me.
Is it possible to achieve this from torch.distributed?

Output:
You can use dist.all_gather to do this:
import torch
import torch.distributed as dist

q = torch.tensor([20, 0, 1, 17]) # generated on each gpu (with different values) as you mentioned
all_q = [torch.zeros_like(q) for _ in range(world_size)] # world_size is the total number of gpu processes you are running. 4 in your case.
all_q = dist.all_gather(all_q, q)

all_q would then have the following:
[torch.tensor([20, 0, 1, 17]), torch.tensor([1, 2, 3, 4]), torch.tensor([5, 6, 7, 8]), torch.tensor([0, 2, 4, 6])]
You can then use torch.cat to collapse all elements into one array if you like.
You can use dist.all_gather_multigpu if you list of lists of tensors.


Instruction: Get frequency of words using Vocab in pytorch Torchtext

Input:
how can i get the frequencies of tokens in a torchtext vocab that is created using build_vocab_from_iterator? link to doc:https://pytorch.org/text/stable/vocab.html#torchtext.vocab.Vocab
def build_vocab(data_iter, tokenizer):
"""Builds vocabulary from iterator"""

    vocab = build_vocab_from_iterator(
            map(tokenizer, data_iter),
            specials=["<unk>"],
            min_freq=MIN_WORD_FREQUENCY,)
    vocab.set_default_index(vocab["<unk>"])
    return vocab

data_iter = get_data_iterator(ds_name, ds_type, data_dir)
tokenizer = get_english_tokenizer()

if not vocab:
    vocab = build_vocab(data_iter, tokenizer)

Output:
You won't be able to get the frequency after you have built the vocab, since that data is lost during the build. It is just checking that the token occurs more than min_freq, and if so, adds it to the vocabulary.
However, you can get the frequency of the tokens before you build the vocabulary. One way to do that is with a Counter (Counter docs):
counter = Counter()
for text in data_iter:
    counter.update(tokenizer(text))

You can get the frequency of the tokens from the counter, then build the vocabulary from the counter:
vocab = vocab.vocab(counter, min_freq=MIN_WORD_FREQUENCY)


Instruction: 2-D Tensor calculated by the mean of 3-D Tensor by specific dimension

Input:
I have a 3-D tensor, with shape (3000, 20, 5). I want to create a 2-D tensor, of shape (3000, 5), using the mean values of the second dimension of the 3-D tensor.
So basically, I want to perform something like:
mean_value = torch.mean(3d_tensor[0][:][0])

But getting values for all values of dimension one a three. I could do a for loop, e.g.:
for j in range(0, 3d_tensor.size()[2]):
    for i in range(0, len(3d_tensor)):

        mean_values[j][i] = torch.mean(3d_tensor[i][:][j])

But this takes a long time to process for large amounts of data.

Output:
You could simply specify axis along which mean should be taken:
mean = torch.mean(tensor, dim=1)

This gives you data of shape (3000, 5)