How to release temporarily consumed GPU memory after each forward?
I have a class like this:

class Stem(nn.Module):

    def __init__(self):
        super(Stem, self).__init__()
        self.out_1 = BasicConv2D(3, 32, kernelSize = 3, stride = 2)
        self.out_2 = BasicConv2D(32, 32, kernelSize = 3, stride = 1)
        self.out_3 = BasicConv2D(32, 64, kernelSize = 3, stride = 1, padding = 1)


    def forward(self, x):
        x = self.out_1(x)
        x = self.out_2(x)
        x = self.out_3(x)

        return x


and the attributes out_1,2,3 of Stem are instances of below class:

class BasicConv2D(nn.Module):

    def __init__(self, inChannels, outChannels, kernelSize, stride, padding = 0):
        super(BasicConv2D, self).__init__()
        self.conv = nn.Conv2d(inChannels, outChannels,
                            kernel_size = kernelSize,
                            stride = stride,
                            padding = padding, bias = False)
        self.bn = nn.BatchNorm2d(outChannels,
                                    eps = 0.001,
                                    momentum = 0.1,
                                    affine = True)
        self.relu = nn.ReLU(inplace = False)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        y = self.relu(x)
        return y


when training, within Stem.forward(), nvidia-smi tells that each line will consume x MBs GPU memory, but after Stem.forward() finished, the memory won't be released, causing the training to quickly crashed, out of GPU memory.

The question is thus:
how to release the temporarily consumed GPU memory?

Your model does look good, so you might want to have a general look at how pytorch manages memory allocation. I suspect that you simply keep pointers to your return value (y) alive (e.g. by accumulating a loss or some such). As pytorch stores the entire attached computation graph, you never free the memory.

See this question and in particular this answer for a more detailed discussion.
