Robust endless loop for server written in Python
Question: I write a server which handles events and uncaught exceptions during handling
the event must not terminate the server.

The server is a single non-threaded python process.

I want to terminate on these errors types:

  * KeyboardInterrupt
  * MemoryError
  * ...

The list of built in exceptions is long:
<https://docs.python.org/2/library/exceptions.html>

I don't want to re-invent this exception handling, since I guess it was done
several times before.

How to proceed?

  1. Have a white-list: A list of exceptions which are ok and processing the next event is the right choice
  2. Have a black-list: A list of exceptions which indicate that terminating the server is the right choice.

Hint: This question is not about running a unix daemon in background. It is
not about double fork and not about redirecting stdin/stdout :-)


Answer: I would do this in a similar way you're thinking of, using the 'you shall not
pass' _[Gandalf](https://www.youtube.com/watch?v=V4UfAL9f74I)_ exception
handler _[`except Exception` to catch all non-system-exiting
exceptions](https://docs.python.org/2.7/library/exceptions.html#exceptions.Exception)_
while creating a black-listed `set` of exceptions that should pass and end be
re-raised.

Using the _Gandalf handler_ will make sure
[`GeneratorExit`](https://docs.python.org/2.7/library/exceptions.html#exceptions.GeneratorExit),
[`SystemExit`](https://docs.python.org/2.7/library/exceptions.html#exceptions.SystemExit)
and
[`KeyboardInterrupt`](https://docs.python.org/2.7/library/exceptions.html#exceptions.KeyboardInterrupt)
(all system-exiting exceptions) pass and terminate the program if no other
handlers are present higher in the call stack. Here is where you can check
with [**`type(e)`**](https://docs.python.org/2.7/library/functions.html#type)
that a `__class__` of a caught exception `e` actually belongs in the set of
black-listed exceptions and
re-[`raise`](https://docs.python.org/2.7/library/exceptions.html#exceptions.KeyboardInterrupt)
it.

As a small demonstration:

    
    
    import exceptions  # Py2.x only
    
    # dictionary holding {exception_name: exception_class}
    excptDict = vars(exceptions)
    
    exceptionNames = ['MemoryError', 'OSError', 'SystemError'] # and others
    
    # set containing black-listed exceptions
    blackSet = {excptDict[exception] for exception in exceptionNames}
    

Now `blackSet = {OSError, SystemError, MemoryError}` holding the classes of
the non-system-exiting exceptions we want to **not** handle.

A `try-except` block can now look like this:

    
    
    try:
        # calls that raise exceptions:
    except Exception as e:
        if type(e) in blackSet: raise e # re-raise
        # else just handle it
    

An **example** which catches all exceptions using
[`BaseException`](https://docs.python.org/2.7/library/exceptions.html#exceptions.BaseException)
can help illustrate what I mean. (this is done _for demonstration purposes
only_ , in order to see how this raising will eventually terminate your
program). **Do note** : _I'm**not** suggesting you use `BaseException`; I'm
using it in order to **demonstrate** what exception will actually 'pass
through' and cause termination_ (i.e everything that `BaseException` catches):

    
    
    for i, j in excptDict.iteritems():
        if i.startswith('__'): continue  # __doc__ and other dunders
        try:
            try:
                raise j
            except Exception as ex:
                # print "Handler 'Exception' caught " + str(i)
                if type(ex) in blackSet:
                    raise ex           
        except BaseException:
            print "Handler 'BaseException' caught " + str(i)
    
    # prints exceptions that would cause the system to exit     
    Handler 'BaseException' caught GeneratorExit
    Handler 'BaseException' caught OSError
    Handler 'BaseException' caught SystemExit
    Handler 'BaseException' caught SystemError
    Handler 'BaseException' caught KeyboardInterrupt
    Handler 'BaseException' caught MemoryError
    Handler 'BaseException' caught BaseException
    

Finally, in order to make this Python 2/3 agnostic, you can `try` and `import
exceptions` and if that fails (which it does in Python 3), fall-back to
importing [`builtins`](https://docs.python.org/3.5/library/builtins.html)
which contains all `Exceptions`; we search the dictionary by name so it makes
no difference:

    
    
    try:
        import exceptions
        excDict = vars(exceptions)
    except ImportError:
        import builtins 
        excDict = vars(builtins)
    

I don't know if there's a smarter way to actually do this, another solution
might be instead of having a `try-except` with a signle `except`, having 2
handlers, one for the black-listed exceptions and the other for the general
case:

    
    
    try:
        # calls that raise exceptions:
    except tuple(blackSet) as be:  # Must go first, of course.
        raise be
    except Exception as e:
        # handle the rest
    

