python multiprocessing/threading cleanup
Question: I have a python tool, that has basically this kind of setup:

    
    
    main process (P1) -> spawns a process (P2) that starts a tcp connection
                      -> spawns a thread (T1) that starts a loop to receive 
                         messages that are sent from P2 to P1 via a Queue (Q1)
    
    server process (P2) -> spawns two threads (T2 and T3) that start loops to
                           receive messages that are sent from P1 to P2 via Queues (Q2 and Q3)
    

The problem I'm having is that when I stop my program (with Ctrl+C), it
doesn't quit. The server process is ended, but the main process just hangs
there and I have to kill it.

The thread loop functions all look the same:

    
    
    def _loop(self):
        while self.running:
            res = self.Q1.get()
            if res is None:
                break
            self._handle_msg(res)
    

All threads are started as daemon:

    
    
    t = Thread(target=self._loop)
    t.setDaemon(True)
    t.start()
    

In my main process, I use atexit, to perform clean-up tasks:

    
    
    atexit.register(self.on_exit)
    

Those clean-up tasks are essentially the following:

1) set `self.running` in P1 to `False` and sent `None` to Q1, so that the
Thread T1 should finish

    
    
    self.running = False
    self.Q1.put(None)
    

2) send a message to P2 via Q2 to inform this process that it is ending

    
    
    self.Q2.put("stop")
    

3) In P2, react to the "stop" message and do what we did in P1

    
    
    self.running = False
    self.Q2.put(None)
    self.Q3.put(None)
    

That is it and in my understanding, that should make everything shut down
nicely, but it doesn't.

The main code of P1 also contains the following endless loop, because
otherwise the program would end prematurely:

    
    
    while running:
        sleep(1)
    

Maybe that has something to do with the problem, but I cannot see why it
should.

So what did I do wrong? Does my setup have major design flaws? Did I forget to
shut down something?

**EDIT**

Ok, I modified my code and managed to make it shut down correctly most of the
time. Unfortunately, from now and then, it still got stuck.

I managed to write a small working example of my code. To demonstrate what
happens, you need to simple start the script and then use `Ctrl + C` to stop
it. It looks like the issue appears now usually if you press `Ctrl + C` as
soon as possible after starting the tool.

    
    
    #!/usr/bin/env python
    # -*- coding: utf-8 -*-
    
    import signal
    import sys
    import logging
    from multiprocessing import Process, Queue
    from threading import Thread
    from time import sleep
    
    
    logger = logging.getLogger("mepy-client")
    
    
    class SocketClientProtocol(object):
    
        def __init__(self, q_in, q_out, q_binary):
            self.q_in = q_in
            self.q_out = q_out
            self.q_binary = q_binary
            self.running = True
            t = Thread(target=self._loop)
            #t.setDaemon(True)
            t.start()
            t = Thread(target=self._loop_binary)
            #t.setDaemon(True)
            t.start()
    
        def _loop(self):
            print "start of loop 2"
            while self.running:
                res = self.q_in.get()
                if res is None:
                    break
                self._handle_msg(res)
            print "end of loop 2"
    
        def _loop_binary(self):
            print "start of loop 3"
            while self.running:
                res = self.q_binary.get()
                if res is None:
                    break
                self._handle_binary(res)
            print "end of loop 3"
    
        def _handle_msg(self, msg):
            msg_type = msg[0]
            if msg_type == "stop2":
                print "STOP RECEIVED"
                self.running = False
                self.q_in.put(None)
                self.q_binary.put(None)
    
        def _put_msg(self, msg):
            self.q_out.put(msg)
    
        def _handle_binary(self, data):
            pass
    
        def handle_element(self):
            self._put_msg(["something"])
    
    def run_twisted(q_in, q_out, q_binary):
        s = SocketClientProtocol(q_in, q_out, q_binary)
        while s.running:
            sleep(2)
            s.handle_element()
    
    
    class MediatorSender(object):
    
        def __init__(self):
            self.q_in = None
            self.q_out = None
            self.q_binary = None
            self.p = None
            self.running = False
    
        def start(self):
            if self.running:
                return
            self.running = True
            self.q_in = Queue()
            self.q_out = Queue()
            self.q_binary = Queue()
            print "!!!!START"
            self.p = Process(target=run_twisted, args=(self.q_in, self.q_out, self.q_binary))
            self.p.start()
            t = Thread(target=self._loop)
            #t.setDaemon(True)
            t.start()
    
        def stop(self):
            print "!!!!STOP"
            if not self.running:
                return
            print "STOP2"
            self.running = False
            self.q_out.put(None)
            self.q_in.put(["stop2"])
            #self.q_in.put(None)
            #self.q_binary.put(None)
    
            try:
                if self.p and self.p.is_alive():
                    self.p.terminate()
            except:
                pass
    
        def _loop(self):
            print "start of loop 1"
            while self.running:
                res = self.q_out.get()
                if res is None:
                    break
                self._handle_msg(res)
            print "end of loop 1"
    
        def _handle_msg(self, msg):
            self._put_msg(msg)
    
        def _put_msg(self, msg):
            self.q_in.put(msg)
    
        def _put_binary(self, msg):
            self.q_binary.put(msg)
    
        def send_chunk(self, chunk):
            self._put_binary(chunk)
    
    running = True
    def signal_handler(signal, frame):
        global running
        if running:
            running = False
            ms.stop()
        else:
            sys.exit(0)
    
    if __name__ == "__main__":
        signal.signal(signal.SIGINT, signal_handler)
        ms = MediatorSender()
        ms.start()
        for i in range(100):
            ms.send_chunk("some chunk of data")
        while running:
            sleep(1)
    


Answer: Maybe you should try to capture `SIGINT` signal, which is generated by `Ctrl +
C` using `signal.signal` like this:

    
    
    #!/usr/bin/env python
    import signal
    import sys
    def signal_handler(signal, frame):
            print('You pressed Ctrl+C!')
            sys.exit(0)
    signal.signal(signal.SIGINT, signal_handler)
    print('Press Ctrl+C')
    signal.pause()
    

Code stolen from [here](http://stackoverflow.com/questions/1112343/how-do-i-
capture-sigint-in-python)

