title: How can one use the StorageStreamDownloader to stream download from a blob and stream upload to a different blob?
body: <p>I believe I have a very simple requirement for which a solution has befuddled me. I am new to the <a href="https://github.com/Azure/azure-sdk-for-python" rel="nofollow noreferrer">azure-python-sdk</a> and have had little success with its new blob streaming functionality.</p>
  <h2>Some context</h2>
  <p>I have used the Java SDK for several years now. Each <a href="https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.torage.blob.cloudblockblob?view=azure-java-legacy" rel="nofollow noreferrer">CloudBlockBlob</a> object has a <a href="https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.storage.blob.blobinputstream?view=azure-java-legacy" rel="nofollow noreferrer">BlobInputStream</a> and a <a href="https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.storage.blob.bloboutputstream?view=azure-java-legacy" rel="nofollow noreferrer">BlobOutputStream</a> object. When a <code>BlobInputStream</code> is opened, one can invoke its many functions (most notably its <a href="https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.storage.blob.blobinputstream.read?view=azure-java-legacy#com_microsoft_azure_storage_blob_BlobInputStream_read__" rel="nofollow noreferrer">read()</a> function) to retrieve data in a true-streaming fashion. A <code>BlobOutputStream</code>, once retrieved, has a <a href="https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.storage.blob.bloboutputstream.write?view=azure-java-legacy#com_microsoft_azure_storage_blob_BlobOutputStream_write_final_byte____" rel="nofollow noreferrer">write(byte[] data)</a> function where one can continuously write data as frequently as they want until the <a href="https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.storage.blob.bloboutputstream.close?view=azure-java-legacy#com_microsoft_azure_storage_blob_BlobOutputStream_close__" rel="nofollow noreferrer">close()</a> function is invoked. So, it was very easy for me to:</p>
  <ol>
  <li>Get a <code>CloudBlockBlob</code> object, open it's <code>BlobInputStream</code> and essentially get back an <code>InputStream</code> that was 'tied' to the <code>CloudBlockBlob</code>. It usually maintained 4MB of data - at least, that's what I understood. When some amount of data is read from its buffer, a new (same amount) of data is introduced, so it always has approximately 4MB of new data (until all data is retrieved).</li>
  <li>Perform some operations on that data.</li>
  <li>Retrieve the <code>CloudBlockBlob</code> object that I am uploading to, get it's <code>BlobOutputStream</code>, and write to it the data I did some operations on.</li>
  </ol>
  <p>A good example of this is if I wanted to compress a file. I had a <code>GzipStreamReader</code> class that would accept an <code>BlobInputStream</code> and an <code>BlobOutputStream</code>. It would read data from the <code>BlobInputStream</code> and, whenever it has compressed some amount of data, write to the <code>BlobOutputStream</code>. It could call write() as many times as it wished; when it finishes reading all the daya, it would close both Input and Output streams, and all was good.</p>
  <h2>Now for Python</h2>
  <p>Now, the Python SDK is a little different, and obviously for good reason; the <code>io</code> module works differently than Java's <code>InputStream</code> and <code>OutputStream</code> classes (which the <code>Blob{Input/Output}Stream</code> classes inherit from. I have been struggling to understand how streaming truly works in Azure's python SDK. To start out, I am just trying to see how the <a href="https://docs.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.storagestreamdownloader?view=azure-python" rel="nofollow noreferrer">StorageStreamDownloader</a> class works. It seems like the <code>StorageStreamDownloader</code> is what holds the 'connection' to the <code>BlockBlob</code> object I am reading data from. If I want to put the data in a stream, I would make a new <code>io.BytesIO()</code> and pass that stream to the <code>StorageStreamDownloader</code>'s <a href="https://docs.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.storagestreamdownloader?view=azure-python#readinto-stream-" rel="nofollow noreferrer">readinto</a> method.</p>
  <p>For uploads, I would call the <a href="https://docs.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.blobclient?view=azure-python" rel="nofollow noreferrer">BlobClient</a>'s <a href="https://docs.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.blobclient?view=azure-python#upload-blob-data--blob-type--blobtype-blockblob---blockblob----length-none--metadata-none----kwargs-" rel="nofollow noreferrer">upload method</a>. The upload method accepts a <code>data</code> parameter that is of type <code>Union[Iterable[AnyStr], IO[AnyStr]]</code>.</p>
  <p>I don't want to go into too much detail about what I understand, because what I understand and what I have done have gotten me nowhere. I am suspicious that I am expecting something that only the Java SDK offers. But, overall, here are the problems I am having:</p>
  <ol>
  <li>When I call <a href="https://docs.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.blobclient?view=azure-python#download-blob-offset-none--length-none----kwargs-" rel="nofollow noreferrer">download_blob</a>, I get back a <code>StorageStreamDownloader</code> with all the data in the blob. Some investigation has shown that I can use the <code>offset</code> and <code>length</code> to download the amount of data I want. Perhaps I can call it once with a <code>download_blob(offset=0, length=4MB)</code>, process the data I get back, then again call <code>download_bloc(offset=4MB, length=4MB)</code>, process the data, etc. This is unfavorable. The other thing I could do is utilize the <code>max_chunk_get_size</code> parameter for the <code>BlobClient</code> and turn on the <code>validate_content</code> flag (make it true) so that the <code>StorageStreamDownloader</code> only downloads 4mb. But this all results in several problems: that's not really streaming from a <code>stream</code> object. I'll still have to call <code>download</code> and <code>readinto</code> several times. And fine, I would do that, if it weren't for the second problem:</li>
  <li>How the heck do I stream an upload? The upload can take a stream. But if the stream doesn't auto-update itself, then I can only upload once, because all the blobs I deal with <strong>must</strong> be <code>BlockBlobs</code>. The docs for the <code>upload_function</code> function say that I can provide a param <code>overwrite</code> that does:</li>
  </ol>
  <blockquote>
  <p>keyword bool overwrite: Whether the blob to be uploaded should overwrite the current data.
  If True, upload_blob will overwrite the existing data. If set to False, the
  operation will fail with ResourceExistsError. The exception to the above is with Append
  blob types: if set to False and the data already exists, an error will not be raised
  and the data will be appended to the existing blob. If set overwrite=True, then the existing
  append blob will be deleted, and a new one created. Defaults to False.</p>
  </blockquote>
  <ol start="2">
  <li>And this makes sense because <code>BlockBlobs</code>, once written to, cannot be written to again. So AFAIK, you can't 'stream' an upload. If I can't have a stream object that is directly tied to the blob, or holds all the data, then the upload() function will terminate as soon as it finishes, right?</li>
  </ol>
  <p>Okay. I am certain I am missing something important. I am also somewhat ignorant when it comes to the <code>io</code> module in Python. Though I have developed in Python for a long time, I never really had to deal with that module too closely. I am sure I am missing something, because this functionality is very basic and exists in all the other azure SDKs I know about.</p>
  <h2>To recap</h2>
  <p>Everything I said above can honestly be ignored, and only this portion read; I am just trying to show I've done some due diligence. I want to know how to stream data from a blob, process the data I get in a stream, then upload that data. I cannot be receiving <em>all</em> the data in a blob at once. Blobs are likely to be over 1GB and all that pretty stuff. I would honestly love some example code that shows:</p>
  <ol>
  <li>Retrieving some data from a blob (the data received in one call should not be more than 10MB) in a stream.</li>
  <li>Compressing the data in that stream.</li>
  <li>Upload the data to a blob.</li>
  </ol>
  <p>This should work for blobs of all sizes; whether its 1MB or 10MB or 10GB should not matter. Step 2 can be anything really; it can also be nothing. Just as long as long as data is being downloaded, inserted into a stream, then uploaded, that would be great. Of course, the other extremely important constraint is that the data per 'download' shouldn't be an amount more than 10MB.</p>
  <p>I hope this makes sense! I just want to stream data. This shouldn't be that hard.</p>
  <p>Edit:</p>
  <p>Some people may want to close this and claim the question is a duplicate. I have forgotten to include something very important: I am currently using the <em>newest</em>, mot up-to-date azure-sdk version. My <code>azure-storage-blob</code> package's version is <code>12.5.0</code>. There have been other questions similar to what I have asked for <strong>severely</strong> outdated versions. I have searched for other answers, but haven't found any for <code>12+</code> versions.</p>