How-To - Update Live Running Python Application
Question: I have a python application , to be more precise a Network Application that
can't go down this means i can't kill the PID since it actually talks with
other servers and clients and so on ... many € per minute of downtime , you
know the usual 24/7 system.

Anyway in my hobby projects i also work a lot with WSGI frameworks and i
noticed that i have the same problem even during off-peak hours.

Anyway imagine a normal server using TCP/UDP ( put here your favourite
WSGI/SIP/Classified Information Server/etc).

Now you perform a git pull in the remote server and there goes the new python
files into the server (these files will of course ONLY affect the data
processing and not the actual sockets so there is no need to re-raise the
sockets or touch in any way the network part).

I don't usually use File monitors since i prefer to use SIGNAL to wakeup the
internal app updater.

Now imagine the following code

    
    
    from mysuper.app import handler
    
    while True:
      data = socket.recv()
      if data:
    
        socket.send(handler(data))
    

Lets imagine that handler is a APP with DB connections, cache connections ,
etc.

What is the best way to update the handler.

Is it safe to call reload(handler) ?

Will this break DB connections ?

Will DB Connections survive to this restart ?

Will current transactions be lost ?

Will this create anti-matter ?

What is the best-pratice patterns that you guys usually use if there are any ?


Answer: It's safe to call `reload(handler)`.

Depends where you initialize your connections. If you make the connections
inside handler(), then yes, they'll be garbage collected when the `handler()`
object falls out of scope. But you wouldn't be connecting inside your main
loop, would you? I'd highly recommend something like:

    
    
    dbconnection = connect(...)
    while True:
        ...
        socket.send(handler(data, dbconnection))
    

if for no other reason than that you won't be making an expensive connection
inside a tight loop.

That said, I'd recommend going with an entirely different architecture. Make a
listener process that does basically nothing more than listen for UDP
datagrams, sends them to a messaging queue like
[RabbitMQ](http://www.rabbitmq.com/), then waits for the reply message to send
the results back to the client. Then write your actual servers that get their
requests from the messaging queue, process them, and send a reply message
back.

If you want to upgrade the UDP server, launch the new instance listening on
another port. Update your firewall rules to redirect incoming traffic to the
new port. Reload the rules. Kill the old process. Voila: seamless cutover.

The real win is from uncoupling your backend. Since multiple processes can
listen for the same messages from your frontend "proxy" service, you can run
several in parallel - on different machines, if you want to. To upgrade the
backend, start a new instance then kill the old one so that there's no time
when at least one instance isn't running.

To scale your proxy, have multiple instances running on different ports or
different hosts, and configure your firewall to randomly redirect incoming
datagrams to one of the proxies.

To scale your backend, run more instances.

