Out of memory error during evaluation but training works fine
I have recently upgraded PyTorch from 0.2 to 0.3. Surprisingly my old programs are throwing an out of memory error during evaluation (in eval() mode) but training works just fine. I am using the same batch size for training and evaluation. I am totally clueless what is happening? Did anyone face similar issue? Is there any possible solution?



I tried using volatile=True param on the variables and it didn't help. Please note, I am not doing anything special to use cuDNN. I am using the default setting.

def validate(self, dev_corpus):
    # Turn on evaluation mode which disables dropout.
    self.model.eval()

    dev_batches = helper.batchify(dev_corpus.data, self.config.batch_size)
    print('number of dev batches = ', len(dev_batches))

    dev_loss = 0
    num_batches = len(dev_batches)
    for batch_no in range(1, num_batches + 1):
        session_queries, session_query_length, rel_docs, rel_docs_length, doc_labels = helper.session_to_tensor(
            dev_batches[batch_no - 1], self.dictionary)
        if self.config.cuda:
            session_queries = session_queries.cuda()
            session_query_length = session_query_length.cuda()
            rel_docs = rel_docs.cuda()
            rel_docs_length = rel_docs_length.cuda()
            doc_labels = doc_labels.cuda()

        loss = self.model(session_queries, session_query_length, rel_docs, rel_docs_length, doc_labels)
        if loss.size(0) &gt; 1:
            loss = loss.mean()
        dev_loss += loss.data[0]

    return dev_loss / num_batches


I am using the above function for evaluation. Here, session_queries, session_query_length, .... rest variables are created by enabling volatile=True.

Please help!!

I think it fails during validation because the volatile flag is now deprecated and has no effect. Starting from 0.4.0, to avoid the gradient being computed for all variables during validation, you should use a context manager. Code example:

with torch.no_grad():
    # Your validation code


With that, the operation history and gradients are not stored. Thus this will save memory.

Refer to 0.4.0 migration guide for more details.

Furthermore, you could delete references to those variables after you finished the evaluation like this:

someVar = Variable(someVar)
del someVar

