title: Process a list of million files on Airflow in parallel
body: <p>I have an ETL pipeline which consists of the following tasks:</p>
  
  <ol>
  <li>crawl a website to collect urls to csv files</li>
  <li>download all the files from url's of Step-1 </li>
  <li>process each file so downloaded in Step-2</li>
  </ol>
  
  <p>I have a python function for each of these steps, let's call them <code>{f1,f2,f3}</code></p>
  
  <p>With the <code>multiprocessing</code> module, and <code>Pool.map</code> call, I can fire <code>n processes</code> for both <code>f2, f3</code></p>
  
  <p>I was converting this entire script into an Airflow DAG. My question is assuming, a scale of million files, I would like to especially in Step-3 use my entire cluster of Celery workers to process the files in parallel. </p>
  
  <p>For example, let's say I have 100 celery worker nodes, with each node having 2 cores which gives me 200 cores- I would like to process say, at least 100 files in parallel.</p>
  
  <p>How do I do this?</p>