Python RLock IO-Bound?
Question: I have a set of CPU-bound processes that take any number of cores to 100%
utilization as long as their only synchronization is getting jobs out of a
Queue.

As soon as I add an RLock to avoid worst case scenarios when updating a
directory in the file system, CPU/core utilization drops to 60%, as if the
processes had become IO-bound.

What's the explanation?

This is not about overall speed. It is about CPU/core utilization, so Python
2/3, Cython, or PyPy should not matter.

**Update:** I gave a partial answer to my own question. The final solution for
my particular case consisted on modifying the way the file system was accessed
so no synchronization was needed (a "sort of" map/reduce).


Answer: It all depends on how `multiprocessing` has implemented `RLock`. I am aware
that multiprocessing _can_ work across hosts which implies that
synchronisation primitives may work across sockets. If that is true, it would
introduce a lot of (variable) latency.

So I did an experiment.

Here's a noddy example of `RLock` being used by more than one process (to
prevent any fast-path where all locks are within the same process):

    
    
    #!/usr/bin/env python
    import multiprocessing
    from time import sleep
    
    lock = multiprocessing.RLock()
    
    def noop(myname):
        # nonlocal lock
        sleep(0.5)
        print myname, "acquiring lock"
        with lock:
            print myname, "has lock"
            sleep(0.5)
        print myname, "released lock"
    
    sProc1 = multiprocessing.Process(target=noop, args=('alice',))
    sProc2 = multiprocessing.Process(target=noop, args=('bob',))
    
    sProc1.start()
    sProc2.start()
    
    sProc1.join()
    sProc2.join()
    

When this is run, its output looks something like this:

    
    
    alice acquiring lock
    alice has lock
    bob acquiring lock
    alice released lock
    bob has lock
    bob released lock
    

Great, so now run it with system call tracing via
[strace](http://en.wikipedia.org/wiki/Strace).

In the command below, the `-ff` option tells the tool to "follow `fork()`"
calls, i.e. trace any processes started by the main one. For reasons of
brevity I'm also using `-e trace=futex,write`, which filters output based on
conclusions I made before posting this. Normally you would run without the
`-e` option and use a text editor / `grep` to explore what happened after the
fact.

    
    
    # strace -ff -e trace=futex,write ./traceme.py
    futex(0x7fffeafe29bc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7fb92ac6c700) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0x7fb92a8540b0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
    futex(0x7fb92aa7131c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
    write(3, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 32) = 32
    Process 25873 attached
    Process 25874 attached
    Process 25872 suspended
    [pid 25873] write(1, "alice acquiring lock\n", 21alice acquiring lock
    ) = 21
    [pid 25873] write(1, "alice has lock\n", 15alice has lock
    ) = 15
    [pid 25874] write(1, "bob acquiring lock\n", 19bob acquiring lock
    ) = 19
    [pid 25874] futex(0x7fb92ac91000, FUTEX_WAIT, 0, NULL <unfinished ...>
    [pid 25873] futex(0x7fb92ac91000, FUTEX_WAKE, 1 <unfinished ...>
    [pid 25874] <... futex resumed> )       = 0
    [pid 25873] <... futex resumed> )       = 1
    [pid 25874] write(1, "bob has lock\n", 13 <unfinished ...>
    bob has lock
    [pid 25873] write(1, "alice released lock\n", 20 <unfinished ...>
    alice released lock
    [pid 25874] <... write resumed> )       = 13
    [pid 25873] <... write resumed> )       = 20
    Process 25872 resumed
    Process 25873 detached
    [pid 25872] --- SIGCHLD (Child exited) @ 0 (0) ---
    Process 25872 suspended
    [pid 25874] write(1, "bob released lock\n", 18bob released lock
    ) = 18
    Process 25872 resumed
    Process 25874 detached
    --- SIGCHLD (Child exited) @ 0 (0) ---
    

From the pattern of print (`write()`) messages and
[`futex`](http://en.wikipedia.org/wiki/Futex) calls which block and later
resume, it seems clear that `RLock` is implemented using `futex`, or "Fast
Userspace Mutex". As the name implies this is a good choice for
synchronisation.

When a process is blocked in a system-call like `futex` the process is
blocking on I/O for all intents and purposes.

All this implies that `multiprocessing.RLock` is efficient and doing what it
was designed to do. So if your application's performance is less than you
expect when using synchronisation, chances are that your algorithm is to
blame.

