title: producer/consumer work queues
body: <p>I'm wrestling with the best way to implement my processing pipeline.</p>
  
  <p>My producers feed work to a BlockingQueue.  On the consumer side, I poll the queue, wrap what I get in a Runnable task, and submit it to an ExecutorService.</p>
  
  <pre><code>while (!isStopping())
  {
      String work = workQueue.poll(1000L, TimeUnit.MILLISECONDS);
      if (work == null)
      {
          break;
      }
      executorService.execute(new Worker(work));   // needs to block if no threads!
  }
  </code></pre>
  
  <p>This is not ideal; the ExecutorService has its own queue, of course, so what's really happening is that I'm always fully draining my work queue and filling the task queue, which slowly empties as the tasks complete.</p>
  
  <p>I realize that I could queue tasks at the producer end, but I'd really rather not do that - I like the indirection/isolation of my work queue being dumb strings; it really isn't any business of the producer what's going to happen to them.  Forcing the producer to queue a Runnable or Callable breaks an abstraction, IMHO.</p>
  
  <p>But I do want the shared work queue to represent the current processing state.  I want to be able to block the producers if the consumers aren't keeping up.</p>
  
  <p>I'd love to use Executors, but I feel like I'm fighting their design.  Can I partially drink the Kool-ade, or do I have to gulp it?  Am I being wrong-headed in resisting queueing tasks?  (I suspect I could set up ThreadPoolExecutor to use a 1-task queue and override it's execute method to block rather than reject-on-queue-full, but that feels gross.)</p>
  
  <p>Suggestions?</p>