Python for loop using Threading or multiprocessing
Question: All, I am rather new and am looking for assistance. I need to perform a string
search on a data set that compressed is about 20 GB of data. I have an eight
core ubuntu box with 32 GB of RAM that I can use to crunch through this but am
not able to implement nor determine the best possible code for such a task.
Would Threading or multiprocessing be best for such a task? Please provide
code samples. Thank you. Please see my current code;

    
    
    #!/usr/bin/python
    import sys
    logs = []
    iplist = []
    
    logs = open(sys.argv[1], 'r').readlines()
    iplist = open(sys.argv[2], 'r').readlines()
    print "+Loaded {0} entries for {1}".format(len(logs), sys.argv[1])
    print "+Loaded {0} entries for {1}".format(len(iplist), sys.argv[2])
    
    for a in logs:
        for b in iplist:
            if a.lower().strip() in b.lower().strip()
                print "Match! --> {0}".format(a.lower().strip())
    


Answer: I'm not sure if multithreading can help you, but your code has a problem that
is bad for performance: Reading the logs in one go consumes incredible amounts
of RAM and thrashes your cache. Instead, open it and read it sequentially,
after all you are making a sequential scan, don't you? Then, don't repeat any
operations on the same data. In particular, the iplist doesn't change, but for
every log entry, you are repeatedly calling `b.lower().strip()`. Do that once,
after reading the file with the IP addresses.

In short, this looks like this:

    
    
    with open(..) as f:
        iplist = [l.lower().strip() for l in f]
    
    with open(..) as f:
        for l in f:
            l = l.lower().strip()
            if l in iplist:
                print('match!')
    

You can improve performance even more by using a set for iplist, because
looking things up there will be faster when there are many elements. That
said, I'm assuming that the second file is huge, while iplist will remain
relatively small.

BTW: You could improve performance with multiple CPUs by using one to read the
file and the other to scan for matches, but I guess the above will already
give you a sufficient performance boost.

