Python: azure-storage-blob takes 6 min to download a 20MB file to local
Did you find your answer? Without a code snippet it would be hard to diagnos your problem. Azure Blob Storage supports read in parallel. If you have BatchServiceClient service that would be straightforward. Otherwise, try Multiprocessing module. Kr.
I have to download .parqeat file from Azure storage and have to analyze the Data.
So I started using python codes with azure-storage-blob library and Pandas to store DataFrames and flask to create API on response.
But now file size in the Azure storage is getting increased up to 30MB(a day file) and as tested it takes 6min to download 20MB file into my local which I can't afford because I have to analyse files of each days in a month, which will have 20 MB each. That's almost 3020MB size.
Any suggestion?