Increasing SpaCy max NLP limit I'm getting this error:

[E088] Text of length 1029371 exceeds maximum of 1000000. The v2.x parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the `nlp.max_length` limit. The limit is in number of characters, so you can check whether your inputs are too long by checking `len(text)`.


The weird thing is that if I reduce the amount of documents being lemmatized, it still says the length exceeds 1 million. Is there a way of increasing the limit past 1 million? The error seems to suggest there is but I'm unable to do so.
Try to raise the nlp.max_length parameter (as your error message suggests):
nlp = spacy.load('en_core_web_sm') 
nlp.max_length = 1500000 #or any large value, as long as you don't run out of RAM

also, when calling your spaCy pipeline, you can disable RAM-hungry intensive parts of the pipeline that are not needed for lemmatization:
doc = nlp(&quot;The sentences we'd like to do lemmatization on&quot;, disable = ['ner', 'parser'])

Finally you should get the results you expect with:
print([x.lemma_ for x in doc])