MemoryError in Python
Question: I have text file, its size is 300 MB. I want to read it and then print 50 most
frequently used words. When i run the program it gives me MemoryError. My code
is as under:-

    
    
    import sys, string 
    import codecs 
    import re
    from collections import Counter
    import collections
    import itertools
    import csv
    import re
    import unicodedata
    
    
    words_1800 = []
    
    with open('E:\\Book\\1800.txt', "r", encoding='ISO-8859-1') as File_1800:
       for line in File_1800:
           sepFile_1800 = line.lower()
           words_1800.extend(re.findall('\w+', sepFile_1800))
    
    
    for wrd_1800 in [words_1800]:
           long_1800=[w for w in words_1800 if len(w)>3]
           common_words_1800 = dict(Counter(long_1800).most_common(50))
    
    print(common_words_1800)
    

It give me the following error:-

    
    
    Traceback (most recent call last):
    File "C:\Python34\CommonWords.py", line 17, in <module>
    words_1800.extend(re.findall('\w+', sepFile_1800))
    MemoryError
    


Answer: You can use a [generator container](http://scottmoonen.com/2008/02/01/python-
generators-saving-time-and-memory/) instead of a list to store the result of
`re.findall` which is much optimized in terms of memory use, you can also use
`re.finditer` instead of `findall` which returns an iterator.

    
    
    with open('E:\\Book\\1800.txt', "r", encoding='ISO-8859-1') as File_1800:
           words_1800=(re.findall('\w+', line.lower()) for line in File_1800)
    

Then the `words_1800` will be an iterator contain lists of founded words or
use

    
    
    with open('E:\\Book\\1800.txt', "r", encoding='ISO-8859-1') as File_1800:
           words_1800=(re.finditer('\w+', line.lower()) for line in File_1800)
    

to get an iterator contains iterators.

