Django: communicate with TCP server (with twisted?)
Question: I have a django application, that needs to talk to a remote TCP server. This
server will send packages and depending on what the package is, I need add
entries to the database and inform other parts of the application. I also need
to actively send requests to the TCP server, for instance when the user
navigates to a certain page, I want to subscribe to a certain stream on the
TCP server. So communication in both directions need to work.

So far, I use the following solution:

I wrote a custom Django command, that I can start with

    
    
    python manage.py listen
    

This command will start a twisted socket server with `reactor.connectTCP(IP,
PORT, factory)` and since it is a django command, I will have access to the
database and all the other parts of my application.

But since I also want to be able to send something to the TCP server,
triggered by a certain django view, I have an additional socket server, that
starts within my twisted application by `reactor.listenTCP(PORT,
server_factory)`.

To this server, I will then connect directly in my django application, within
a new thread:

    
    
    class MSocket:
    
        def __init__(self):
            self.stopped = False
            self.socket = None
            self.queue = []
            self.process = start_new_thread(self.__connect__, ())
            atexit.register(self.terminate)
    
    
        def terminate(self):
            self.stopped = True
            try:
                self.socket.close()
            except:
                pass
    
        def __connect__(self):
            if self.stopped:
                return
            attempts = 0
            self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            while True and not self.stopped:
                try:
                    print "Connecting to Socket Server..."
                    self.socket.connect(("127.0.0.1", settings.SOCKET_PORT))
                    print "Connection Successful!"
                    for msg in self.queue:
                        self.socket.send(msg)
                    self.queue = []
                    break
                except:
                    pause = min(int(round(1.2**attempts)), 30)
                    print "Connection Failed! Try again in " + str(pause) + " seconds."
                    sleep(pause)
                    attempts += 1
    
            self.__loop__()
    
        def __loop__(self):
            if self.stopped:
                return
            while True and not self.stopped:
                try:
                    data = self.socket.recv(1024)
                except:
                    try:
                        self.socket.close()
                    except:
                        pass
                    break
                if not data:
                    break
            self.__connect__()
    
        def send(self, msg):
            try:
                self.socket.send(msg)
                return True
            except:
                self.queue.append(msg)
                return False
    
    
    m_socket = MSocket()
    

`m_socket` will then be imported by the main `urls.py` so that it starts with
django.

So my setup looks kind this:

Sending to TCP Server:

    
    
    Django (connect:8001) ------->  (listen:8001) Twisted (connect:4444) ------> (listen:4444) TCP-Server
    

Receiving from TCP Server

    
    
    TCP-Server (listen:4444) ------> (connect:4444) Twisted ---(direct access)---> Django
    

It all seems to work that way, but I fear that this is not a really good
solution, since I have to open this extra TCP connection. So my question would
be now, if the setup can be optimized (and I'm sure it can) and how it can be
done.


Answer: This is not going to work unless you monkey patch Django (as mentioned by
@pss) I had a similar situation so this is what I did.

  1. Run a separate twisted deamon.
  2. To communicate from Django to Twisted, use Unix sockets. The local twisted server can listen on Unix sockets (AF_UNIX) and Django can simply connect to that socket. This will avoid going through the TCP stack
  3. To communicate from Twisted to Django, you have multiple options, a) call Django url with the data b) launch a script (Django management command) c) Use celery to launch the the above Django command d) Use a queue (zeromq or rabbit) and have your Django management command listen in on the queue (preferred)

With the last option, you get much better throughput, durability and it scales
well.

