title: How to parallelize meta ingestion job in Amundsen?
body: <p>I am trying to parallelize meta ingestion job in my project where I am using <strong>Amundsen</strong> but I have facing issues. Below is the code snippet for the same. I am performing <strong>parallelization at Account level</strong> in Snowflake, the metadata that I am getting from Snowflake is ingested to <strong>Neo4J</strong> then.</p>
  <pre><code>def process_all_snowflake_accounts():
      &quot;&quot;&quot;Function that loops through all the SF accounts&quot;&quot;&quot;
      snowflake_config = read_snowflake_configuration()
      start_time = time.time()
      processes = []
      for ac_key, ac_config in snowflake_config.items():
          process = multiprocessing.Process(target=multiprocessing_snowflake_accounts, args=(ac_key, ac_config))
          processes.append(process)
          process.start()
      for process in processes:
          process.join()
      print(&quot;CPU Unit: &quot;, multiprocessing.cpu_count)
      print('****************************************************************')
      print('Total time taken: ', time.time() - start_time)
      print('****************************************************************')
  </code></pre>
  <p>Few time the above code is just skipping some database of accounts and not showing any error, but most of the time it is showing the error mentioned below:</p>
  <pre><code>&quot;Scanning Snowflake ...&quot;
  &quot;Process account: Account-1&quot;
  &quot;Process account: Account-2&quot;
  &quot;Process account: Account-3&quot;
  &quot;Launching job for Account-1-DB-1&quot;
  &quot;Launching job for Account-2-DB-1&quot;
  &quot;Launching job for Account-3-DB-1&quot;
  &quot;Launching job for Account-2-DB-2&quot;
  &quot;Launching job for Account-1-DB-2&quot;
  ERROR:databuilder.publisher.neo4j_csv_publisher:Failed to publish. Rolling back.
  Traceback (most recent call last):
    File &quot;/usr/local/lib/python3.7/site-packages/amundsen_databuilder-2.6.4-py3.7.egg/databuilder/publisher/neo4j_csv_publisher.py&quot;, line 202, in publish_impl
      tx = self._publish_node(node_file, tx=tx)
    File &quot;/usr/local/lib/python3.7/site-packages/amundsen_databuilder-2.6.4-py3.7.egg/databuilder/publisher/neo4j_csv_publisher.py&quot;, line 266, in _publish_node
      with open(node_file, 'r', encoding='utf8') as node_csv:
  FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/amundsen/tables/nodes/Description_4.csv'
  </code></pre>