python 2.6.x theading / signals /atexit fail on some versions?
Question: I've seen a lot of questions related to this... but my code _works_ on python
2.6.2 and _fails_ to work on python 2.6.5. Am I wrong in thinking that the
whole atexit "functions registered via this module are not called when the
program is killed by a signal" thing shouldn't count here because I'm catching
the signal and then exiting cleanly? What's going on here? Whats the proper
way to do this?

    
    
    import atexit, sys, signal, time, threading
    
    terminate = False
    threads = []
    
    def test_loop():
        while True:
            if terminate:
                print('stopping thread')
                break
            else:
                print('looping')
                time.sleep(1)
    
    @atexit.register
    def shutdown():
        global terminate
        print('shutdown detected')
        terminate = True
        for thread in threads:
            thread.join()
    
    def close_handler(signum, frame):
        print('caught signal')
        sys.exit(0)
    
    def run():
        global threads
        thread = threading.Thread(target=test_loop)
        thread.start()
        threads.append(thread)
    
        while True:
            time.sleep(2)
            print('main')
    
    signal.signal(signal.SIGINT, close_handler)
    
    if __name__ == "__main__":
        run()
    

python 2.6.2:

    
    
    $ python halp.py 
    looping
    looping
    looping
    main
    looping
    main
    looping
    looping
    looping
    main
    looping
    ^Ccaught signal
    shutdown detected
    stopping thread
    

python 2.6.5:

    
    
    $ python halp.py 
    looping
    looping
    looping
    main
    looping
    looping
    main
    looping
    looping
    main
    ^Ccaught signal
    looping
    looping
    looping
    looping
    ...
    looping
    looping
    Killed <- kill -9 process at this point
    

The main thread on 2.6.5 appears to never execute the atexit functions.


Answer: The root difference here is actually unrelated to both signals and atexit, but
rather a change in the behavior of `sys.exit`.

Before around 2.6.5, `sys.exit` (more accurately, SystemExit being caught at
the top level) would cause the interpreter to exit; if threads were still
running, they'd be terminated, just as with POSIX threads.

Around 2.6.5, the behavior changed: the effect of `sys.exit` is now
essentially the same as returning from the main function of the program. When
you do _that_ \--in both versions--the interpreter waits for all threads to be
joined before exiting.

The relevant change is that `Py_Finalize` now calls
`wait_for_thread_shutdown()` near the top, where it didn't before.

This behavioral change seems incorrect, primarily because it no longer
functions as documented, which is simply: "Exit from Python." The practical
effect is no longer to exit from Python, but simply to exit the thread. (As a
side note, `sys.exit` has never exited Python when called from another thread,
but that obscure divergance from documented behavior doesn't justify a much
bigger one.)

I can see the appeal of the new behavior: rather than two ways to exit the
main thread ("exit and wait for threads" and "exit immediately"), there's only
one, as sys.exit is essentially identical to simply returning from the top
function. However, it's a breaking change and diverges from documented
behavior, which far outweighs that.

Because of this change, after `sys.exit` from the signal handler above, the
interpreter sits around waiting for threads to exit and then runs `atexit`
handlers after they do. Since it's the handler itself that tells the threads
to exit, the result is a deadlock.

