What is causing the allocation of 12 GB of memory and causing CUDA out of memory error? Model, Data or something else?
I am trying to build a 3D CNN based video classifier using Pytorch. When i try to run a single datapoint i run into this error:
CUDA out of memory. Tried to allocate 1.20 GiB (GPU 0; 14.76 GiB total capacity; 12.60 GiB already allocated; 1.09 GiB free; 12.61 GiB reserved in total by PyTorch)

My data of 1000 videos has a size of around 90MB on disk. I think i am only loading model and data onto GPU. I am not able to understand what is causing the already allocated 12.6 GiB of memory? is my model too big or it has something to do with data loaded or something else? Here is a snippet of my model. It is based on C3D by Tran et al, 2015.
    self.conv1 = nn.Conv3d(3, 64, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool1 = nn.MaxPool3d(kernel_size=(1, 2, 2), stride=(1, 2, 2))

    self.conv2 = nn.Conv3d(64, 128, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool2 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))

    self.conv3a = nn.Conv3d(128, 256, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.conv3b = nn.Conv3d(256, 256, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool3 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))

    self.conv4a = nn.Conv3d(256, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.conv4b = nn.Conv3d(512, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool4 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))

    self.conv5a = nn.Conv3d(512, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.conv5b = nn.Conv3d(512, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool5 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2), padding=(0, 1, 1))

    self.fc6 = nn.Linear(373248, 4096)
    self.fc7 = nn.Linear(4096, 128)
    self.fc8 = nn.Linear(128, 2)

    self.dropout = nn.Dropout(p=0.5)

    self.relu = nn.ReLU()
    self.softmax = nn.Softmax()

Also, i am using Dataset, DataLoader and DeviceDataLoader to load the data. Even if they load all the data at once, what is causing to inflate 90MB of data to 12.6GiB.

It's important to note that the real reason you have out of memory issues most of the time is not necessarily the inherent size of the model itself (though it is directly related to this). Even 100,000,000 parameters, with single floating point precision, only takes about 1 GB to store. (Now, in your case, your model actually is so large that just your parameters may consume on the order of 100GB of memory, but in most models this wouldn't be the  case).
Generally, you'll have memory issues when the forward pass through the model computes and stores the gradient of each parameter, with respect to the previous layer's parameters. This can temporarily double (or more) the number of floating point values being stored in memory, so the combined total of your model, your data, and the gradient/computation graph exceed the available GPU memory.
As suggested above, your solutions are

reduce your batch size
downsample your data (if possible)
reduce the complexity of your model

(As an aside, I'm not totally sure how you are intending to train this model but you cannot possibly hope to feed a whole video to your model as input. Aside from the memory issues, it would be extremely difficult for the model to learn the temporal relationships necessary to make sense of the data. Conventional video processing techniques would probably pass a single frame or a few frames at a time to a model and use a transformer, lstm, or other sequence-to-output model to learn the temporal context.)
