title: spring boot API - document processing and executing python script on documents in parallel
body: <p><strong>Scenario</strong>:</p>
  
  <ol>
  <li>In my application, there are 3 processes which are copying documents on a shared drive in their respective folders.</li>
  <li><p>As soon as any document is copied on shared drive (by any process), directory watcher (Java) code picks up the document and call the Python script using "Process" and do some processing on the document. code snippet is as follows:</p>
  
  <pre><code>Process pr = Runtime.getRuntime().exec(pythonCommand);
              // retrieve output from python script
              BufferedReader bfr = new BufferedReader(new InputStreamReader(pr.getInputStream()));
              String line = "";
              while ((line = bfr.readLine()) != null) {
                  // display each output line from python script
                  logger.info(line);
              }
              pr.waitFor();
  </code></pre></li>
  <li><p>Currently my code waits till python code execution is completed on the document. Only after that it pick up the next document. <strong>Python code takes 30 secs to complete</strong>. </p></li>
  <li>After processing the document, document is moved from the current folder to archive OR error folder. </li>
  <li>Please find below screen shot of the scenario:
  <a href="https://i.stack.imgur.com/CzlO4.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/CzlO4.jpg" alt="enter image description here"></a></li>
  </ol>
  
  <p><strong>What is the problem?</strong></p>
  
  <ol>
  <li>My code is processing documents in sequential manner and I need to process the document in parallel.</li>
  <li>As Python code takes around 30 seconds, some of the events created by directory watcher are also getting lost. </li>
  <li>If around 400 documents are coming within a short span of time, document processing stops.</li>
  </ol>
  
  <p><strong>What I am looking for?</strong></p>
  
  <ol>
  <li>Design solution for processing documents in parallel.</li>
  <li>In case of any failure scenario for document processing, pending documents must be processed automatically.</li>
  <li>I tried spring boot schedular as well but still documents are getting processed in sequential manner only. </li>
  <li>Is it possible to call the Python code in parallel as a background process.</li>
  </ol>
  
  <p>Sorry for the long question but I am stuck at this from many days and already looked many similar questions. 
  Thank you!</p>