title: What is the best way scale out work to multiple machines?
body: <p>We're developing a .NET app that must make up to tens of thousands of small webservice calls to a 3rd party webservice.  We would prefer a more 'chunky' call, but the 3rd party does not support it.  We've designed the client to use a configurable number of worker threads, and through testing have code that is fairly well optimized for one multicore machine.  However, we still want to improve the speed, and are looking at spreading the work accross multiple machines. We're well versed in typical client/server/database apps, but new to designing for multiple machines.  So, a few questions related to that:</p>
  
  <ul>
  <li>Is there any other client-side optimization, besides multithreading, that we should look at that could improve speed of a http request/response? (I should note this is a non-standard webservice, so is implemented using WebClient, not a WCF or SOAP client)</li>
  <li>Our current thinking is to use WCF to publish chunks of work to MSMQ, and run clients on one or more machines to pull work off of the queue.  We have experience with WCF + MSMQ, but want to be sure we're not missing better options.  Are there other, better ways to do this today?</li>
  <li>I've seen some 3rd party tools like DigiPede and Microsoft's HPC offerings, but these seem like overkill.  Any experience with those products or reasons we should consider them over roll-our-own?</li>
  </ul>