How to utilize all GPUs when dealing with pytorch code?
I have 2 GPUs and when I am working with pytorch code, only one GPU is used. I tried CUDA_VISIBLE_DEVICES=0,1 python xxx.py, but occurs 


  'CUDA_VISIBLE_DEVICES: command not found'


problems. I have also tried to add the following lines in object py file:

 import os    
    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"    
    os.environ["CUDA_VISIBLE_DEVICES"] = "0,1" 


but still only one GPU is utilized.  

You need to parallelize the training data to each GPU seperatly. Data Parallelism is implemented using torch.nn.DataParallel. An example from the pytorch documentation:

import torch
import torch.nn as nn


class DataParallelModel(nn.Module):

    def __init__(self):
        super().__init__()
        self.block1 = nn.Linear(10, 20)

        # wrap block2 in DataParallel
        self.block2 = nn.Linear(20, 20)
        self.block2 = nn.DataParallel(self.block2)

        self.block3 = nn.Linear(20, 20)

    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        return x

