Spacy inference goes OOM when processing several documents
I'm using spacy to process documents that come through rest api. To be more specific, I'm using transformer based model en_core_web_trf for NER, running on GPU. Here is a code snippet of the spacy related class (It is packed inside some basic flask server and but I don't suppose that matters here)
class SpacyExtractor():
    def __init__(self):
        spacy.require_gpu()
        self.model = spacy.load('en_core_web_trf',
                                 disable=[&quot;tagger&quot;, &quot;parser&quot;, &quot;attribute_ruler&quot;, &quot;lemmatizer&quot;])


    def get_named_entities(self, text: str):
        doc = self.model(text)
        entities = []
        for ent in doc.ents:
            entities.append((ent.text, ent.label_))
        return entities

The problem is, with each call of get_named_entities, the amount of GPU memory allocated goes up. And it is like 2-3 GB every time (I checked this by repeatedly calling nvidia-smi while the app was processing the docs). So after a few calls, I get OOM error
RuntimeError: CUDA out of memory. Tried to allocate 2.35 GiB (GPU 0; 10.76 GiB total capacity; 5.02 GiB already allocated; 1.18 GiB free; 8.41 GiB reserved in total by PyTorch)
Documents are not huge at all, 1-100 pages of text for each one.
I think I make some mistake, but I just don't see it.
Environment: Ubuntu 18.04, Python 3.8, spacy 3.1.3, cuda 9.1, RTX 2080Ti 11GB RAM
EDIT: Also, I found out the OOM error when processing a single really long document, presented as a single long string.


The problem is, with each call of get_named_entities, the amount of GPU memory allocated goes up.

You should detach your data as explained in the FAQ:

Don’t accumulate history across your training loop. By default,
computations involving variables that require gradients will keep
history. This means that you should avoid using such variables in
computations which will live beyond your training loops, e.g., when
tracking statistics. Instead, you should detach the variable or access
its underlying data.


Edit
You can also use
with torch.no_grad():
    doc = self.model(text)


EDIT: Also, I found out the OOM error when processing a single really long document, presented as a single long string.

Well, this is to be expected.
