Threading in python - processing multiple large files concurrently
Question: I'm new to python and I'm having trouble understanding how threading works. By
skimming through the documentation, my understanding is that calling `join()`
on a thread is the recommended way of blocking until it completes.

To give a bit of background, I have 48 large csv files (multiple GB) which I
am trying to parse in order to find inconsistencies. The threads share no
state. This can be done single threadedly in a reasonable ammount of time for
a one-off, but I am trying to do it concurrently as an exercise.

Here's a skeleton of the file processing:

    
    
    def process_file(data_file):
      with open(data_file) as f:
        print "Start processing {0}".format(data_file)
        line = f.readline()
        while line:
          # logic omitted for brevity; can post if required
          # pretty certain it works as expected, single 'thread' works fine
          line = f.readline()
    
      print "Finished processing file {0} with {1} errors".format(data_file, error_count)
    
    def process_file_callable(data_file):
      try:
        process_file(data_file)
      except:
        print >> sys.stderr, "Error processing file {0}".format(data_file)
    

And the concurrent bit:

    
    
    def partition_list(l, n):
        """ Yield successive n-sized partitions from a list.
        """
        for i in xrange(0, len(l), n):
            yield l[i:i+n]
    
    partitions = list(partition_list(data_files, 4))
    for partition in partitions:
      threads = []
      for data_file in partition:
        print "Processing file {0}".format(data_file)
        t = Thread(name=data_file, target=process_file_callable, args = (data_file,))
        threads.append(t)
        t.start()
    
      for t in threads:
        print "Joining {0}".format(t.getName())
        t.join(5)
    
      print "Joined the first chunk of {0}".format(map(lambda t: t.getName(), threads))
    

I run this as:

    
    
    python -u datautils/cleaner.py > cleaner.out 2> cleaner.err
    

My understanding is that join() should block the calling thread waiting for
the thread it's called on to finish, however the behaviour I'm observing is
inconsistent with my expectation.

I never see errors in the error file, but I also never see the expected log
messages on stdout.

The parent process does not terminate unless I explicitly kill it from the
shell. If I check how many prints I have for `Finished ...` it's never the
expected 48, but somewhere between 12 and 15. However, having run this single-
threadedly, I can confirm that the multithreaded run is actually processing
everything and doing all the expected validation, only it does not seem to
terminate cleanly.

I know I must be doing something wrong, but I would really appreciate if you
can point me in the right direction.


Answer: I can't understand where mistake in your code. But I can recommend you to
refactor it a little bit. First at all, threading in python is not concurrent
at all. It's just illusion, because there is a [Global Interpreter
Lock](https://wiki.python.org/moin/GlobalInterpreterLock), so only one thread
can be executed in same time. That's why I recommend you to use
[multiprocessing
module](https://docs.python.org/2/library/multiprocessing.html):

    
    
    from multiprocessing import Pool, cpu_count
    pool = Pool(cpu_count)
    for partition in partition_list(data_files, 4):
        res = pool.map(process_file_callable, partition)
        print res
    

At second, you are using not pythonic way to read file:

    
    
    with open(...) as f:
       line = f.readline()
        while line:
           ... # do(line)
          line = f.readline()
    

Here is pythonic way:

    
    
    with open(...) as f:
        for line in f:
             ... # do(line)
    

> This is memory efficient, fast, and leads to simple code. (c) PyDoc

By the way, I have only one hypothesis what can happen with your program in
multithreading way - app became more slower, because unordered access to hard
disk drive is significantly slower than ordered. You can try to check this
hypothesis using `iostat` or `htop`, if you are using Linux.

If your app does not finish work, and it doesn't do anything in process
monitor (cpu or disk is not active), it means you have some kind of deadlock
or blocked access to same resource.

