title: Boost ASIO as an event loop with boost lockfree queue for socket write
body: <p>I am using boost ASIO for a TCP client. for the most part the ASIO is a glorified event loop for read and write. There is actually only one client managed by the ASIO.</p>
  
  <p>The architecture is like this - 
  The TCP server streams continuous messages. The Client will read the messages, process it and ack back with proper code.</p>
  
  <p>My code runs in client side. There is one thread running io_service. The io_service thread reads messages and distributes it to N number of worker threads using a boost lockfree SPSC queue. The workers after processing posts the replies to the io_service thread.</p>
  
  <p>most important concern for me is the rate of read and write. So I am using synchronous reads and writes. </p>
  
  <p>Read Code: </p>
  
  <pre><code> void read ()
   {
  
       if (_connected &amp;&amp; !_readInProgress) {
           _socket.async_read_some(boost::asio::null_buffers(),
                   make_boost_alloc_handler(_readAllocator,
                       [self = shared_from_this(), this] (ErrorType err, unsigned a)
                       {
                            connection()-&gt;handleRead(err);
                            _readInProgress = false;
  
                            if (err) disconnect();
                            else asyncRead();
  
                       });
           _readInProgress = true;
       }
   }
  </code></pre>
  
  <p>Basically I use read_some with nullbuffer() and then directly use Unix system calls to read the messages. The read give N number of messages which are enqueued to threads in a loop. </p>
  
  <p>I want use the boost SPSC queue in the reverse direction for writes to the socket from workers.</p>
  
  <p>Write:</p>
  
  <pre><code>// Get the queue to post writes
   auto getWriteQ ()
   {
       static thread_local auto q = 
           std::make_shared&lt; LFQType &gt;(_epoch);
       return q;
   }
  </code></pre>
  
  <p>So each thread gets a thread-local Q using getWriteQ. The writes to the queue looks like this:</p>
  
  <pre><code>void write (Buf&amp; buf) override
   {
       auto q = getWriteQ();
  
       while (!q-&gt;enqueue(buf) &amp;&amp; _connected);
  
       if (!_connected) return;
  
        _ioService.post( [self = shared_from_this(), this, q]() 
        {
             writeHelper(q); });
        }
  }
  </code></pre>
  
  <p>Now this is inefficent, as we do a ioservice post for each write. The write handler at a time actually writes upto 32 messages in a single system-call using sendmmsg()</p>
  
  <p>So I am looking for help with 2 things:</p>
  
  <ul>
  <li>Is the design any good?</li>
  <li><p>Any fool proof way to minimize the no. of posts. I was thinking of keep an atomic enqueue count. The worker thread will do this -
  the writing thread does this - (Pseudo code)  </p>
  
  <p>bool post = false;</p>
  
  <p>if(enqueue_count == 0)  post = true</p>
  
  <p>// enqueue the message</p>
  
  <p>++enqueue_count</p>
  
  <p>if(post)
      // post the queue event</p></li>
  </ul>
  
  <p>The io-service thread does this - </p>
  
  <pre><code>enqueue_count -= num_processed;
  if (enqueue_count) 
  // repost the queue for further processing
  </code></pre>
  
  <p>Would this work if the enqueue_count is atomic ?</p>