title: How to download 100 million rows from Azure Table Storage FAST
body: <p>I have been tasked with downloading around 100 million rows of data from Azure Table Storage.  The important thing here being speed.</p>
  
  <p>The process we are using is downloading 10,000 rows from Azure Table storage.  Process them into a local instance of Sql Server.  While processing the rows it deletes 100 rows at a time from the Azure table.  This process is threaded to have 8 threads downloading 10,000 rows at a time.</p>
  
  <p>The only problem with this is that according to our calculations.  It will take around 40 days to download and process the around 100 million rows we have stored.  Does anyone know a faster way to accomplish this task?</p>
  
  <p>A side question: During the download process Azure will send back xml that just does not have any data.  It doesn't send back an error.  But it sends this:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="utf-8" standalone="yes"?&gt;
  &lt;feed xml:base="azure-url/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom"&gt;
    &lt;title type="text"&gt;CommandLogTable&lt;/title&gt;
    &lt;id&gt;azure-url/CommandLogTable&lt;/id&gt;
    &lt;updated&gt;2010-07-12T19:50:55Z&lt;/updated&gt;
    &lt;link rel="self" title="CommandLogTable" href="CommandLogTable" /&gt;
  &lt;/feed&gt;
  0
  </code></pre>
  
  <p>Does anyone else have this problem and have a fix for it?</p>