Problems with Threading in Python 2.5, KeyError: 51, Help debugging?
Question: I have a python script which runs a particular script large number of times
(for monte carlo purpose) and the way I have scripted it is that, I queue up
the script the desired number of times it should be run then I spawn threads
and each thread runs the script once and again when its done.

Once the script in a particular thread is finished, the output is written to a
file by accessing a lock (so my guess was that only one thread accesses the
lock at a given time). Once the lock is released by one thread, the next
thread accesses it and adds its output to the previously written file and
rewrites it.

I am not facing a problem when the number of iterations is small like 10 or 20
but when its large like 50 or 150, python returns a KeyError: 51 telling me
element doesn't exist and the error it points out to is within the lock which
puzzles me since only one thread should access the lock at once and I do not
expect an error.

This is the class I use:

    
    
    class errorclass(threading.Thread):
    
        def __init__(self, queue):
            self.__queue=queue
            threading.Thread.__init__(self)
    
        def run(self):
            while 1:
                  item = self.__queue.get()
                  if item is None: break
                  result = myfunction()
                  lock = threading.RLock()
                  lock.acquire()
                  ADD entries from current thread to entries in file and
                  REWRITE FILE
                  lock.release()
    
    queue = Queue.Queue()
    
    for i in range(threads):
        errorclass(queue).start()
    
    for i in range(desired iterations):
        queue.put(i)
    for i in range(threads):
        queue.put(None)
    

Python returns with KeyError: 51 for large number of desired iterations during
the adding/write file operation after lock access, I am wondering if this is
the correct way to use the lock since every thread has a lock operation rather
than every thread accessing a shared lock? What would be the way to rectify
this?


Answer: What you have right now is a _new lock_ for every _iteration_ in each thread's
`run` method. In effect, there is no locking going on at all. If you want to
protect writes to a file, you need to make sure that _all_ threads that access
the same file use the same lock object. The simplest way to do that is to
create it at the global level:

    
    
    lock = threading.RLock()
    class errorclass(...):
        ...
        def run(self)
            ... # get items and process
            with lock:
                ADD entries from current thread to entries in file and REWRITE FILE
    

You'll need to use `from __future__ import with_statement` to use the `with`
statement in Python 2.5, but it makes sure you never forget to unlock the
lock, not even when an error occurs. (If you need Python 2.4-and-earlier
compatibility, you'll have to use a try/finally instead.)

