title: MemoryError: Unable to allocate 1.83 MiB for an array with shape (5004, 96) and data type int32
body: <p>When I want to process a huge csv file I'm getting a MemoryError <code>MemoryError: Unable to allocate 1.83 MiB for an array with shape (5004, 96) and data type int32</code>. The error happens at:</p>
  <pre class="lang-py prettyprint-override"><code>processed_texts = [text for text in nlp.pipe(str(tokenized_texts),
                                               disable=[&quot;ner&quot;,
                                                        &quot;parser&quot;])]
  </code></pre>
  <p>Will this be fixed when I'm using multiple threads? If so, has anybody some examples in Python, because I'm coming from Java..</p>
  <p>Whole script:</p>
  <pre class="lang-py prettyprint-override"><code>df = pd.read_csv('posts_result.csv')
  df_sample = df.sample(frac=0.1, replace=False, random_state=1)
  
  &quot;&quot;&quot; DATA EXPLORATION &quot;&quot;&quot;
  
  text_test = df_sample.post.tolist()
  
  
  # Start the tokenization
  def tokenize_hashtag(text):
      punctuations = '!&quot;$%&amp;\'()*+,-./:;&lt;=&gt;?[\\]^_`{|}~'
      for punctuation in punctuations:
          text = str(text).replace(punctuation, '')
      text = text.lower()
      text = text.split()
      return text
  
  
  tokenized_texts = [tokenize_hashtag(text) for text in text_test]
  
  nlp = spacy.load(&quot;en_core_web_sm&quot;)
  
  processed_texts = [text for text in nlp.pipe(str(tokenized_texts),
                                               disable=[&quot;ner&quot;,
                                                        &quot;parser&quot;])]
  
  df_sample['processed'] = tokenized_texts
  
  tokenized_texts = [[word.text for word in text if (word.pos_ == 'NOUN' or word.pos_ == 'VERB' or word.pos_ == 'PROPN') and not len(word.text) &gt;12 and not word.is_punct and not word.is_stop and not word.text=='X'
                     and not word.text == '@Name']
                      for text in processed_texts]
  </code></pre>