Would it possible to use all memory of GPUs with one model?
There is a model and two GPUs. I put the model on GPU with model.cuda(). If I passed a big image to the model, it allocated all memory of GPU0 and then it raised CUDA out of memory error without allocating any memory of GPU1.

Because there is only one image every forward(), I can not use such torch.nn.DataParallel things to split input.

Is there any way to use all the memory of GPUs when passing a image to the model?

I'm using Python3.7 and Pytorch1.1.

you can split your model into two submodule. 
like this:

class MyModel(nn.Module):
    def __init__(self, split_gpus):
        self.large_submodule1 = ...
        self.large_submodule2 = ...

        self.split_gpus = split_gpus
        if split_gpus:
            self.large_submodule1.cuda(0)
            self.large_submodule1.cuda(1)

    def forward(self, x):
        x = self.large_submodule1(x)
        if split_gpus:
            x = x.cuda(1) # P2P GPU transfer
        return self.large_submodule2(x)


from the pytorch discuss
