Looking for a quick way to speed up my code
Question: I am looking for a way to speed up my code. I managed to speed up most parts
of my code, reducing runtime to about 10 hours, but it's still not fast enough
and since I'm running out of time I'm looking for a quick way to optimize my
code.

**An example:**

    
    
    text = pd.read_csv(os.path.join(dir,"text.csv"),chunksize = 5000)
    new_text = [np.array(chunk)[:,2] for chunk in text]
    new_text = list(itertools.chain.from_iterable(new_text))
    

In the code above I read in about 6 million rows of text documents in chunks
and flatten them. This code takes about 3-4 hours to execute. This is the main
bottleneck of my program. **edit** : I realized that I wasn't very clear on
what the main issue was, **The flattening is the part which takes the most
amount of time.**

Also this part of my program takes a long time:

    
    
        train_dict = dict(izip(text,labels))
        result = [train_dict[test[sample]] if test[sample] in train_dict else predictions[sample] for sample in xrange(len(predictions))]
    

The code above first zips the text documents with their corresponding labels
(This a machine learning task, with the train_dict being the training set).
Earlier in the program I generated predictions on a test set. There are
duplicates between my train and test set so I need to find those duplicates.
Therefore, I need to iterate over my test set row by row (2 million rows in
total), when I find a duplicate I actually don't want to use the predicted
label, but the label from the duplicate in the train_dict. I assign the result
of this iteration to the variable result in the above code.

I heard there are various libraries in python that could speed up parts of
your code, but I don't know which of those could do the job and right know I
do not have the time to investigate this, that is why I need someone to point
me in the right direction. Is there a way with which I could speed the code
snippets above up?

**edit2**

I have investigated again. And it is definitely a memory issue. I tried to
read the file in a row by row manner and after a while the speed declined
dramatically, furthermore my ram usage is nearly 100%, and python's disk usage
increased sharply. How can I decrease the memory footprint? Or should I find a
way to make sure that I don't hold everything into memory?

**edit3** As memory is the main issue of my problems I'll give an outline of a
part of my program. I have dropped the predictions for the time being, which
reduced the complexity of my program significantly, instead I insert a
standard sample for every non duplicate in my test set.

    
    
    import numpy as np
    import pandas as pd
    import itertools
    import os
    
    train = pd.read_csv(os.path.join(dir,"Train.csv"),chunksize = 5000)
    train_2 = pd.read_csv(os.path.join(dir,"Train.csv"),chunksize = 5000)
    test = pd.read_csv(os.path.join(dir,"Test.csv"), chunksize = 80000)
    sample = list(np.array(pd.read_csv(os.path.join(dir,"Samples.csv"))[:,2]))#this file is only 70mb
    sample = sample[1]
    test_set = [np.array(chunk)[:,2] for chunk in test]
    test_set = list(itertools.chain.from_iterable(test_set))
    
    train_set = [np.array(chunk)[:,2] for chunk in train]
    train_set = list(itertools.chain.from_iterable(train_set))
    labels = [np.array(chunk)[:,3] for chunk in train_2]
    labels = list(itertools.chain.from_iterable(labels))
    
    """zipping train and labels"""
    train_dict = dict(izip(train,labels))
    """finding duplicates"""
    results = [train_dict[test[item]] if test[item] in train_dict else sample for item in xrange(len(test))]
    

Although this isn't my entire program, this is the part of my code that needs
optimization. As you can see I am only using three important modules in this
part, pandas, numpy and itertools. The memory issues arise when flattening
train_set and test_set. The only thing I am doing is reading in the files,
getting the necessary parts zipping the train documents with the corresponding
labels in a dictionary. And then search for duplicates.

**edit 4** As requested I'll give an explanation of my data sets. My Train.csv
contains 4 columns. The first columns contain ID's for every sample, the
second column contains titles and the third column contains text body
samples(varying from 100-700 words). The fourth column contains category
labels. Test.csv contains only the ID's and text bodies and titles. The
columns are separated by commas.


Answer: You can try [Cython](http://cython.org/). It supports numpy and can give you a
nice speedup. Here is an introduction and explanation of what needs to be done
<http://www.youtube.com/watch?v=Iw9-GckD-gQ>

