PyTorch new model function leaking gpu memory
I had a perfectly working code, until I had to do some modification to make it more flexible with some hyperparameters.
In the bellow snippet you can see my forward function which before worked using the 2 commented lines. As I needed more flexibility in the nn.Linear from self.classifier, I made the function classifierAdaptive() which does the same thing but allows some parameters to change in the nn.Linear.
Previous GPU Ram usage was around 7GB/11GB, now I am running out of GPU Ram and I can't figure out why. I can't think of any variable to cause memory issues.
Edit:
The dataset is of images of size 2200x2200, from which I make small cuts to train on. These cuts are currently hardcoded as in 6 * 6.
Desired behaviour will be to only have 1 nn.Linear which would work for all img.shapes. However, as @jsho said, the mode maybe is creating too many layers instead of just 1 and overwriting?
I cannot include the entire code for debugging. It would be too much. But if needed can be seen here without the bellow code modification.
    self.classifier = nn.Sequential(
        nn.Linear(512 * 6 * 6, 1024),
        nn.LeakyReLU(0.2, True),
        nn.Linear(1024, 1),
        nn.Sigmoid()
    )

def classifierAdaptive(self,img):
    out = torch.flatten(img, 1)
    outLinear = nn.Linear(512 * img.shape[2] * img.shape[3], 1024).cuda()
    out = outLinear(out).cuda()
    out = self.lrelu(out) # self.lrelu = nn.LeakyReLU(0.2, True)
    out = self.dens2(out) # self.dens2 = nn.Linear(1024, 1)
    out = self.sig(out) # self.sig = nn.Sigmoid()

    return out

def forward(self, x):
    out = self.features(x)

    # out = torch.flatten(out, 1)
    # out = self.classifier(out)

    out = self.classifierAdaptive(out)

    return out


You are creating a new layer for every image with a different size, which not only will not work, but every new layer will have its set of parameters which it saves in GPU memory, and so you will quickly run out of memory space. This is a usual problem in image processing due to the fixed size of linear layers as opposed to the dynamic H,W dimensions of the convolution layers. The only real way to fix it is to reshape your images to the same size prior to putting them in the network (look at  torch.nn.functional.interpolate or cv2.resize) or to use a convolution with a kernel size of 1 (effectively a linear layer) and do some global average/max pooling to shrink the data to the same size.
