I have a CSV with around 10,000,000 lines. Each line looks like:keyword1, keyword2, ....keywordnkeyword1, keyword2, ....keywordnkeyword1, keyword2, ....keywordnkeyword1, keyword2, ....keywordnThe script which does a frequency count and outputs to another text file:from collections import Counterimport csvimport bz2import syscsv.field_size_limit(999999999)all_terms = []#with open('keywords.csv', 'rb') as f:with bz2.BZ2File(keywords.csv.bz2, r) as f:    reader = csv.reader(f)    for idx, row in enumerate(reader):        for term in row:            terms = term.split(,)            for x in terms:                all_terms.append( x.strip().lower() )            if idx % 100000 == 0:                print idxc = Counter(all_terms)with open(counts.txt, 'w+') as fl:    for k,v in  c.most_common():        fl.write( {} | {}\n.format(k,v) )I can't think of a way to minimize the memory utilization using generators and I'm not sure I could do a better job with the frequency counting job than the Counter class from the collections module.
Frequency analysis for counting keywords
python;performance;python 2.7;csv