title: Architecture around processing single and batch requests simultaneously
body: <p>I have a WCF service hosted in a Windows Service. This service exposes 2 methods:</p>
  
  <ol>
  <li><code>bool ProcessClaim(string options, ref string xml);</code> Takes some data as an input, does some processing (including IO-bound operations, like DB queries), and returns the result back.</li>
  <li><code>void RunJob(string ticket);</code> Returns immediately. According to <code>ticket</code>, reads input data from storage (e.g. DB or file system), does the same processing for each data element, and saves result back to storage. Batch usually consists of many claims.</li>
  </ol>
  
  <p>Users can call <code>ProcessClaim</code> to process single requests and <code>RunJob</code> to run batches. Several batches can run simultaneously. Each processing request is wrapped as <code>Task</code>, therefore all requests are executed in parallel.
  The problem was not to allow batches to chock up processing queue by scheduling a lot of requests. Other words, if user executes huge batch, it would block small batches and single processing requests for significant amount of time.
  So I came up with the following schema, well described by <a href="http://www.albahari.com/threading/" rel="nofollow">Albahari</a> (very briefly):</p>
  
  <pre><code>public sealed class ProcessingQueue : IDisposable
  {
      private class WorkItem
      {
          public readonly TaskCompletionSource&lt;string&gt; TaskSource;
          public readonly string Options;
          public readonly string Claim;
          public readonly CancellationToken? CancelToken;
  
          public WorkItem(
              TaskCompletionSource&lt;string&gt; taskSource,
              string options,
              string claim,
              CancellationToken? cancelToken)
          {
              TaskSource = taskSource;
              Options = options;
              Claim = claim;
              CancelToken = cancelToken;
          }
      }
  
      public ProcessingQueue()
          : this(Environment.ProcessorCount)
      {
      }
  
      public ProcessingQueue(int workerCount)
      {
          _taskQ = new BlockingCollection&lt;WorkItem&gt;(workerCount * 2);
  
          for (var i = 0; i &lt; workerCount; i++)
              Task.Factory.StartNew(Consume);
      }
  
      public void Dispose()
      {
          _taskQ.CompleteAdding();
      }
  
      private readonly BlockingCollection&lt;WorkItem&gt; _taskQ;
  
      public Task&lt;string&gt; EnqueueTask(string options, string claim, CancellationToken? cancelToken = null)
      {
          var tcs = new TaskCompletionSource&lt;string&gt;();
          _taskQ.Add(new WorkItem(tcs, options, claim, cancelToken));
          return tcs.Task;
      }
  
      public static Task&lt;string&gt; ProcessRequest(string options, string claim, CancellationToken? cancelToken = null)
      {
          return Task&lt;string&gt;.Factory.StartNew(() =&gt; ProcessItem(options, claim));
      }
  
      private void Consume()
      {
          foreach (var workItem in _taskQ.GetConsumingEnumerable())
          {
              if (workItem.CancelToken.HasValue &amp;&amp; workItem.CancelToken.Value.IsCancellationRequested)
                  workItem.TaskSource.SetCanceled();
              else
              {
                  try
                  {
                      workItem.TaskSource.SetResult(ProcessItem(workItem.Options, workItem.Claim));
                  }
                  catch (Exception ex)
                  {
                      workItem.TaskSource.SetException(ex);
                  }
              }
          }
      }
  
      private static string ProcessItem(string options, string claim)
      {
          // do some actual work here
          Thread.Sleep(2000); // simulate work;
          return options + claim; // return final result
      }
  }
  </code></pre>
  
  <p>Static method <code>ProcessRequest</code> can be used to process single requests, whereas instance method <code>EnqueueTask</code> - for batch processing. Of course all batches must use a single shared instance of <code>ProcessingQueue</code>. Although this approach works pretty good and allows to control the pace of multiple batches running simultaneously, there is something that seems wrong to me:</p>
  
  <ul>
  <li>Have to maintain a pool of working threads manually</li>
  <li>Difficult to guess the optimal number of working threads (I use the number of processor cores by default)</li>
  <li>Bunch of threads remain blocked when no batches are running, wasting system resources</li>
  <li>IO-bound portions of processing block worker threads reducing efficiency of CPU usage</li>
  </ul>
  
  <p>I wonder, is there a better way of dealing with this type of scenarios?</p>
  
  <p><strong>Update:</strong>
  One of the requirements is to provide full power for batches, meaning when user executes one batch, and there are no other incoming requests, all resources must be dedicated towards processing this batch.</p>