how do I diagnose a vanishing port listener?
Question: I'm pulling data off a port using a python process, launched as an upstart job
on an Ubuntu server. The data is sent using TCP with each client sending a
single relatively small string of information:

The upstart config:

    
    
    start on runlevel [2345]
    stop on runlevel [!2345]
    
    respawn
    respawn limit 3 5
    
    setuid takeaim
    setgid takeaim
    
    exec /home/takeaim/production/deploy/production/update_service_demon.sh
    

The update_service_demon.sh script (I found it easier to debug separating this
out of upstart):

    
    
    #!/bin/bash
    
    # Make sure we're in the right virtual env and location
    source /home/takeaim/.virtualenvs/production/bin/activate
    source /home/takeaim/.virtualenvs/production/bin/postactivate
    
    cd /home/takeaim/production
    
    exec python drupdate/dr_update_service.py
    

The python script (it dispatches the real work to a celery worker):

    
    
    from collections import defaultdict
    import select
    import socket
    from django.conf import settings
    
    from drupdate.tasks import do_dr_update
    
    
    def create_server_socket():
        """Set up the and return server socket"""
        server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        server_socket.setblocking(0)
        server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        server_socket.bind(('0.0.0.0', settings.DRUPDATE['PORT']))
        server_socket.listen(settings.DRUPDATE['MAX_CONNECT_REQUESTS'])
        return server_socket
    
    
    def serve(echo_only=False):
    
        message_length = settings.DRUPDATE['MSG_LENGTH']
        message_chunks = defaultdict(list)
        server_socket = create_server_socket()
        inputs = [server_socket]
        while inputs:
            readable, writable, exceptional = select.select(inputs, [], inputs)
            for sock in readable:
                if sock is server_socket:
                    client_socket, address = server_socket.accept()
                    client_socket.setblocking(0)
                    inputs.append(client_socket)
                else:
                    chunk = sock.recv(message_length)
                    if chunk:
                        message_chunks[sock].append(chunk)
                    else:
                        # This client_socket is finished, hand off message for processing
                        message = ''.join(message_chunks[sock])
                        if echo_only:
                            print(message)
                        else:
                            do_dr_update.delay(message)
                        inputs.remove(sock)
                        sock.close()
    
            for sock in exceptional:
                inputs.remove(sock)
                sock.close()
                if sock is server_socket:
                    # replace bad server socket
                    server_socket = create_server_socket()
                    inputs.append(server_socket)
    
    
    if __name__ == "__main__":
        import argparse
        parser = argparse.ArgumentParser(description="Process incoming DR messages")
        parser.add_argument('--echo', help='Just echo incoming messages to the console - no updates will take place',
                            dest='echo_only', action='store_true', default=False)
        args = parser.parse_args()
        serve(echo_only=args.echo_only)
    

The process disappears every now and then despite the restart. I'm reluctant
to make the restarts unlimited unless I can understand why the process
disappears. A manual restart works fine... until it disappears again. It can
be up for days and then just vanishes.

What is the best way to find out what is going on?


Answer: Add enough logging to the system to enable traces to be analysed after a
failure.

Here are some suggestions for logging in order of verbosity:

  * Replace the `exec python drupdate/dr_update_service.py` call with the following snippet which will log the exit code of your python process to syslog on exit. The exit code may give some clues as to how the process terminate. eg If the process terminate by a signal the exit code will be >= 128.

`python drupdate/dr_update_service.py || logger "He's dead Jim, exit code $?"`

  * Add a `try/except` block around your `server` call in `__main__`. In the exception handler, print the traceback to file or a logging subsystem.

  * If the above methods fail to provide clues, wrap your entire script with a call to `strace -f -tt` and divert the output to a log file. This will trace the entire set of system calls made by your program, their arguments and return codes. This will help debug issues which may be related to system calls which return errors. Applying this method will slow down your process and generate a huge amount of output which may in turn change the behaviour of your program and mask the underlying issue.

