Multiprocessing python-server creates too many temp-directories
Question: I'm trying to implement a server in python3.3 that has a separate thread
preloaded to do all the processing for the incoming connections.

    
    
    from multiprocessing import Process, Pipe, Queue
    from multiprocessing.reduction import reduce_socket
    import time
    import socketserver,socket
    
    
    def process(q):
        while 1:
            fn,args = q.get()
            conn = fn(*args)
    
            while conn.recv(1, socket.MSG_PEEK):
                buf = conn.recv(100)
                if not buf: break
                conn.send(b"Got it: ")
                conn.send(buf)
    
            conn.close()
    
    class MyHandler(socketserver.BaseRequestHandler):
        def handle(self):
            print("Opening connection")
            print("Processing")
            self.server.q.put(reduce_socket(self.request))
            while self.request.recv(1, socket.MSG_PEEK):
                time.sleep(1)
            print("Closing connection")
    
    class MyServer(socketserver.ForkingTCPServer):
        p = Process
        q = Queue()
        parent_conn,child_conn = Pipe()
        def __init__(self,server_address,handler):
            socketserver.ForkingTCPServer.__init__(self,server_address, handler)
            self.p = Process(target=process,args=(self.q,))
            self.p.start()
        def __del__(self):
            self.p.join()
    
    
    server_address = ('',9999)
    myserver = MyServer(server_address,MyHandler)
    myserver.serve_forever()
    

I can test that it works using the following script:

    
    
    from multiprocessing.reduction import reduce_socket
    import time
    import socket
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect(('localhost', 9999))
    
    time.sleep(1)
    print("reduce_socket(s)")
    fn,args = reduce_socket(s)
    time.sleep(1)
    print("rebuild_socket(s)")
    conn = fn(*args)
    time.sleep(1)
    print("using_socket(s)")
    
    conn.send("poks")
    print conn.recv(255)
    conn.send("poks")
    print conn.recv(255)
    conn.send("")
    print conn.recv(255)
    conn.close()
    

Unfortunately there seems to be something that is wrong since after running
the test for n times, my tmp-folder is filled with subfolders:

    
    
    $ ls /tmp/pymp*|wc -l
    32000
    

These temporary files are created by `socket_reduce()`. Interestingly the
`rebuild/reduce_socket()` in the client also creates the temporary files, but
they are removed once the function exits. The maximum amount of folders in my
current tmp-filesystem is 32000 which causes a problem. I could remove the
/tmp/pymp*-files by hand or somewhere in the server, but I guess there should
also be the correct way to do this. Can anyone help me with this?


Answer: Okay, Kind of fixed it. From the .`./lib/python3.3/multiprocessing/util.py:`

    
    
    $ grep "def get_temp_dir" -B5 /usr/local/lib/python3.3/multiprocessing/util.py
    
    #
    # Function returning a temp directory which will be removed on exit
    #
    
    def get_temp_dir():
    

It seems that the temporary directory should be available until the process
quits. Since my `process()` and `main()` both run forever, the temporary file
won't be removed. To fix it I can create another process that will hand the
reduced_socket to the `process()`:

    
    
    def process(q):
        while 1:
            fn,args = q.get()
            conn = fn(*args)
    
            while conn.recv(1, socket.MSG_PEEK):
                buf = conn.recv(100)
                if not buf: break
                conn.send(b"Got it: ")
                conn.send(buf)
    
            conn.close()
            q.put("ok")
    
    
    
    class MyHandler(socketserver.BaseRequestHandler):
        def socket_to_process(self,q):
            q.put(reduce_socket(self.request))
            q.get()
        def handle(self):
            p = Process(target=self.socket_to_process,args=(self.server.q,))
            p.start()
            p.join()
    

This way the temporary file is created in a subprocess that will exit once the
`process()` has done its thing with the input. I don't think this is an
elegant way of doing it but it works. If someone knows better, please let
stackoverflow know.

