anchor.diff: -Rational: Or why am I bothering to rewrite nanomsg?
  -===================================================
  += Rational: Or why am I bothering to rewrite nanomsg?
   Garrett D'Amore <garrett@damore.org>
  -v0.1, October 18, 2017
  +v0.2, February 22, 2018
   
   
   NOTE: You might want to review
  
positive.code: from one process to another you may have to execute up to 6 extra system
  calls, beyond the 2 required to actually send and receive the message.
  
  There are cases where this file descriptor logic is easier for existing
  applications to integrate into event loops (e.g. they already have a thread
  blocked in poll().)
  
  But for many cases this is not necessary.  A simple callback mechanism
  would be far better, with the FDs available only as an option for code
  that needs them.  This is the approach that we have taken with nng.
  
  As another consequence of our approach, we do not require file descriptors
  for sockets at all, so it is possible to create applications containing
  many thousands of inproc sockets with no files open at all.  (Obviously
  if you're going to perform real I/O to other processes or other systems,
  you're going to need to have the underlying transport file descriptors
  open.)
  
  
  POSIX APIs
  ----------
  
  Another of Martin's goals, which seems worthwhile at first, was the
  attempt to provide a familiar BSD POSIX API.  As a C programmer, this
  really attracted me.
  
  The problem is that the POSIX APIs are actually really horrible.  In
  particular the semantics around cmsg are about as arcane and painful as
  one can imagine.  Largely, this has meant that extensions to the cmsg
  API simply have not occurred in nanomsg.
  
  The cmsg API specified by POSIX is as bad as it is because POSIX had
  requirements not to break APIs that already existed, and they needed to
  shim something that would work with existing implementations, including
  getting across a system call boundary.  nanomsg  has never had such
  constraints.
  
  Attempting to retain low numbered "socket descriptors" had its own
  problems -- a huge source of use-after-close bugs, which made the
  use of nn_close() incredibly dangerous for multithreaded sockets.
  (If one thread closes and opens a new socket, other threads still using
  the old socket might wind up accessing the "new" socket without realizing
  it.)
  
  The other thing is that BSD socket APIs are super familiar to UNIX C
  programmers -- but experience with nanomsg has taught us already that these
  are actually in the minority of nanomsg's users.  Most of our users are
  coming to us from C++ (object oriented), Java, and Python backgrounds.
  For them the BSD sockets API is frankly somewhat bizarre and alien.
  
  With nng, we realized that constraining ourselves to the mistakes of the
  POSIX API was hurting rather than helping. So nng provides a much friendlier
  interface for getting properties associated with messages.
  
  In nng we also generally try hard to avoid reusing
  an identifier until no other option exists.  This generally means most
  applications won't see socket reuse until billions of other sockets
  have been opened.
  
  
  Compatibility
  -------------
  
  Of course, there are a number of existing nanomsg consumers "in the wild"
  already.  It is important to continue to support them.  So I decided from
  the get go to implement a "compatibility" layer, that provides the same
  API, and as much as possible the same ABI, as legacy nanomsg.  However,
  new features and capabilities would not necessarily be exposed to the
  the legacy API.
  
  Today nng offers this.  You can relink an existing nanomsg binary against
  nng instead of nanomsg, and it usually Just Works.  Source compatibility
  is almost as easy, although the application code needs to be modified
  to use different header files.  (Note: I am considering changing this
  in the future.)
  
  
  Asynchronous IO
  ---------------
  
  As a consequence of our experience with threads being so unscalable,
  we decided to create a new underlying abstraction modeled largely on
  Windows IO completion ports.  (As bad as so many of the Windows APIs
  are, the IO completion port stuff is actually pretty nice.)  Under the
  hood in nng all I/O is asynchronous, and we have `struct nni_aio` objects
  for each pending I/O.  These have an associated completion routine.
  
  The completion routines are _usually_ run on a separate worker thread
  
negative.code: perhaps on multiple cores, but nanomsg itself runs all socket code on
  a single worker thread.)
  
  There is another problem too -- the `inproc` code that moves messages
  between one socket and another was incredibly racy.  This is because the
  two sockets have different locks, and so dealing with the different
  contexts was tricky (and consequently buggy).  (I've since, I think, fixed
  the worst of the bugs here, but only after many hours of pulling out hair.)
  
  The state machines also make fairly linear flow really difficult to follow.
  For example, there is a state machine to read the header information.  This
  