title: Executing azcopy sync command through python concurrent.futures
body: <p>I am trying to sync almost 1000 azure storage containers from one storage account to another using <code>azcopy sync</code> command. Since the number of containers is more, I wrote a python script to run azcopy command concurrently. Execution works fine. I also want to generate a CSV report at the end of the run. But I am facing an issue with report generation. Here is the code I have,</p>
  <pre><code>#!/usr/bin/python
  import subprocess, os,re, csv
  from subprocess import Popen, PIPE
  import concurrent.futures
  from datetime import datetime
  
  container_filename=&quot;list.txt&quot;
  container_list = []
  source_storage_account_url=&quot;&quot;
  dest_storage_account_url=&quot;&quot;
  source_sas_token=&quot;&quot;
  dest_sas_token=&quot;&quot;
  os.environ['AZCOPY_CONCURRENCY_VALUE'] = '500'
  
  header = ['container', 'Number of File Transfers', 'Total Number of Transfer', 'Number of Transfers Completed', 'Number of Transfers Failed', 'Number of Transfers Skipped', 'TotalBytesTransferred', 'Final Job Status']
  f = open(&quot;sync_report.csv&quot;, &quot;a&quot;, encoding='UTF8')
  csvwriter = csv.writer(f)
  csvwriter.writerow(header)
  
  now = datetime.now()
  current_time = now.strftime(&quot;%H:%M:%S&quot;)
  print(&quot;Current Time =&quot;, current_time)
  
  def execute_azcopy_command(command):
      cmd=command[1]
      container=command[0]
      result=&quot;&quot;
      print(cmd)
      try:
          result=subprocess.check_output(
              cmd, stderr=subprocess.STDOUT, shell=True, timeout=360,
              universal_newlines=True)
          source_scan_pattern          =  re.compile(r'Files Scanned at Source: \d+')
          target_scan_pattern          =  re.compile(r'Files Scanned at Destination: \d+')
          copy_transfer_pattern        =  re.compile(r'Total Number Of Copy Transfers: \d+')
          completed_transfer_pattern   =  re.compile(r'Number of Copy Transfers Completed: \d+')
          failed_transfer_pattern      =  re.compile(r'Number of Copy Transfers Failed: \d+')
          bytes_transfer_pattern       =  re.compile(r'Total Number of Bytes Transferred: \d+')
          enumerated_transfer_pattern  =  re.compile(r'Total Number of Bytes Enumerated: \d+')
          job_status_pattern           =  re.compile(r'Final Job Status: .*')
  
          source_scan                  = source_scan_pattern.search(result)
          target_scan                  = target_scan_pattern.search(result)
          copy_transfer                = copy_transfer_pattern.search(result)
          completed_transfer           = completed_transfer_pattern.search(result)
          failed_transfer              = failed_transfer_pattern.search(result)
          bytes_transfer               = bytes_transfer_pattern.search(result)
          enumerated_transfer          = enumerated_transfer_pattern.search(result)
          status                       =  job_status_pattern.search(result)
          
          csvwriter.writerow([container, source_scan.group().split(':')[1].strip(), target_scan.group().split(':')[1].strip(),  copy_transfer.group().split(':')[1].strip(), completed_transfer.group().split(':')[1].strip(), failed_transfer.group().split(':')[1].strip(),  bytes_transfer.group().split(':')[1].strip(), enumerated_transfer.group().split(':')[1].strip(), status.group().split(':')[1].strip()])
      except subprocess.CalledProcessError as exec:
          print(&quot;command failed with error code &quot; , exec.returncode , &quot; and message &quot; + exec.output)
          return &quot;&quot;
      else:
          return result
  
  with open(container_filename, &quot;r&quot;) as a_file:
      for line in a_file:
          stripped_line = line.strip()
          container_list.append(f&quot;{stripped_line}&quot;)
  
  with concurrent.futures.ThreadPoolExecutor() as executor:
      futures = []
      for c_id in container_list:
          source=f&quot;{source_storage_account_url}/{c_id}/?{source_sas_token}&quot;
          target=f&quot;{dest_storage_account_url}/{c_id}/?{dest_sas_token}&quot;
          command=f&quot;/usr/local/bin/azcopy sync \&quot;{source}\&quot; \&quot;{target}\&quot; --recursive --log-level=ERROR&quot;
          cmd = (c_id, command)
          futures.append(executor.submit(execute_azcopy_command, command=cmd))
      for future in concurrent.futures.as_completed(futures):
          print(future.result())
  
  now = datetime.now()
  current_time = now.strftime(&quot;%H:%M:%S&quot;)
  print(&quot;Current Time =&quot;, current_time)
  </code></pre>
  <p>With the above code, the issue I am facing is, though all the containers get syncd successfully, csv report does not have report of all the containers.
  Does anyone know how to solve this issue ? or is there a better way of handling this scenario ? any suggestions will be appreciated.</p>
  <p>Thanks,
  Tintu</p>