Neural Network on GPU using system RAM instead of GPU memory
I built a basic chatbot using PyTorch, and in the training code, I moved both the neural network as well as the training data to the gpu. However, when I run the program, it uses up to 2GB of my ram. There is a little gpu memory that is used, but not that much. When i run the same program, but this time on the cpu, it takes only about 900mb of my ram.
Can anyone tell me why is is happening? i have attached my code, as well as a couple of screenshots.
Sorry if the answer is pretty obvious, I'm new to deep learning.
My code:
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
from nltk_utils import tokenizer, stem, bag_of_words
import pandas as pd

device = torch.device(&quot;cuda&quot;)

#Neural Network
class NeuralNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.l1 = nn.Linear(input_size, hidden_size)
        self.l2 = nn.Linear(hidden_size, hidden_size)
        self.l3 = nn.Linear(hidden_size, hidden_size)
        self.l4 = nn.Linear(hidden_size, num_classes)
        self.relu = nn.ReLU()
        
    def forward(self, x):
        out = self.relu(self.l1(x))
        out = self.relu(self.l2(out))
        out = self.relu(self.l3(out))
        out = self.l4(out)
        return out
    
#data initialization
data = pd.read_csv(&quot;path_to_train_data&quot;)
data = data.dropna(axis=0)

allwords = []
taglist = []
xy = []
ignorewords = [&quot;?&quot;, &quot;!&quot;, &quot;'&quot;, &quot;,&quot;]

taglist.extend(x for x in data[&quot;tag&quot;] if x not in taglist)

#developing vocabulary
for x in data[&quot;pattern&quot;]:
    w = tokenizer(x)
    allwords.extend(stem(y) for y in w if y not in ignorewords)
    
#making training data
for indx, x in enumerate(data[&quot;pattern&quot;]):
    w = tokenizer(x)
    bag = bag_of_words(w, allwords)
    tag = taglist.index(data[&quot;tag&quot;].iloc[indx])
    xy.append((bag, tag))

xtrain = np.array([x[0] for x in xy])
ytrain = np.array([x[1] for x in xy])

class TestDataset(Dataset):
    def __init__(self):
        self.num_classes = len(xtrain)
        self.xdata = torch.from_numpy(xtrain.astype(np.float32))
        self.ydata = torch.from_numpy(ytrain.astype(np.float32))

    def __getitem__(self, index):
        return self.xdata[index], self.ydata[index]
    
    def __len__(self):
        return self.num_classes

dataset = TestDataset()    
train_data = DataLoader(dataset=dataset,
                        batch_size=8,
                        shuffle=True,
                        num_workers=0,
                        pin_memory=True)

inputSize = len(xtrain[0])
hiddenSize = 32
outputSize = len(taglist)

model = NeuralNetwork(inputSize, hiddenSize, outputSize).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters())

epochs = 5000

for epoch in range(epochs):
    for (words, labels) in train_data:
        words = words.to(device)
        labels = labels.type(torch.LongTensor)
        labels = labels.to(device)

        y_pred = model(words)
        
        loss = criterion(y_pred, labels)
        
        optimizer.zero_grad(set_to_none=True)
        loss.backward()
        optimizer.step()
        
    with torch.no_grad():
        if (epoch + 1) % 10 == 0:
            print(f&quot;epoch: {epoch + 1}, loss: {loss:.30f}&quot;)

While running on GPU:
GPU utilization,
RAM utilization
While running on CPU:
RAM utilization

RAM is the place where your data gets stacked to be proceed, the stacks are then transferred to SRAM or otherwise know as cache which is the nearest memory to CPU (referred as host). The general rule to select the memory of RAM for a system is considered as equivalent or greater than GPU memory.
For Example: If the GPU memory is 8GB you need to have a RAM of 8GB or greater to ensure optimal performance.
So, when running on GPU's (referred as Device) your data is directly transferred to GPU to perform tensor operations. But on CPU it can not perform parallel computations thus using lesser memory of RAM.
You can try it out with different batch_sizes and observe the RAM usage. Note that if your batch of data is not fitting in GPU memory, you will be seeing an CUDA out of memory error.
