title: Is there such a thing as too many threads?
body: <p>Environment: Ubuntu 16.04 - Linux , compiling C++11 using GCC. Software doesn't need to be cross-platform - but efficient in it's task, and be an excellent daemon.</p>
  
  <p>I have a somewhat simple application at the moment, which basically acts as an intermediary between a third-party service and a websocket connection. so users connect to my service via websocket to talk to said third-party service. </p>
  
  <pre><code>| End-User |  &lt;-C1-&gt; | My Application | &lt;-C2-&gt; | 3rd Party Service |
  </code></pre>
  
  <p>My applicaiton is threaded with 2 main threads currently:</p>
  
  <ol>
  <li><p>Thread 1 - Listens to websocket connection, whenever it gets a message, it pushes to a fifo task queue an object that contains the message and the websocket connection that requested said message.</p></li>
  <li><p>Thread 2 - Loops through the queue of message, and pops out a message and processes it</p></li>
  </ol>
  
  <p>The issue is that Thread 1 is pretty fast, and can easily handle 100s of websocket connections. Thread 2 does blocking tasks at times and can be slow, since certain queue items processing takes a while by said third party service. This means that if a User A does request 1, which takes 5 seconds to respond, than User B who came afterwards and make request 2, will have to wait for user A's request 1 to finish, even if request 2 takes less than 1ms.</p>
  
  <p>My proposed solution for this is to have:</p>
  
  <ul>
  <li>Thread 1 - WebSocket connection listener</li>
  <li>Thread 2 Task delegator</li>
  <li>Thread 3 - 100 - Task workers </li>
  </ul>
  
  <p>Thread 1 can handle easily 100s of websocket connections, and each connection can make requests of task that can take anywhere between 1ms - 1 minute. all the thread 3 - 100 are sleeping. The reason for this many threads is because if there are 50-60 connections which all make different long-running requests, than each of these time consuming calls only block a single thread, the other threads are still free to work on the queue, and do other tasks. </p>
  
  <p>I know switching threads is an intensive operation, but i am not aware of any other approach than mulithreading here.</p>
  
  <p>I have solved the same issue with a single thread - but the problem was that the server stopped processing any websocket messages when it was blocking by waiting for the third party service. So i bumped it up to two threads - one for websocket, and the other for handling the task queue. But now the problem is a single worker on the task queue is slow since it is sequentially handling blocking IO operations.</p>
  
  <p>Does this sound like a terrible design idea? Any thoughts on best practices?</p>