title: Preventing of taking too many memory in python multiprocessing
body: <p>I have I/O processes in real time, so i can not use Pool functionality. My script want to process income files (about 5-40 minutes) parallel. I can receive n+1 files in one second, but as well i can receive 1 file for 24h. <strong>How can i prevent multiprocessing for taking too much memory on my server? I have multiple scripts working in background, also i do not want to someone use this vunerability.</strong></p>
  <p>Code:</p>
  <pre><code>
  def on_created(event):
      try:
          proc = multiprocessing.Process(target=x_content_check, args=(event.src_path,))
          proc.start()
          proc.join(timeout=TIMEOUT_TIME)
          proc.terminate() 
  
  
  if __name__ == &quot;__main__&quot;:
      try:
          patterns = [&quot;*&quot;]
          ignore_patterns = None
          ignore_directories = False
          case_sensitive = True
          my_event_handler = PatternMatchingEventHandler(patterns, ignore_patterns, ignore_directories, case_sensitive)
          my_event_handler.on_created = on_created
          go_recursively = False
          my_observer = Observer()
          my_observer.schedule(my_event_handler, path, recursive=go_recursively)
          my_observer.start()
          try:
              while True:
                  time.sleep(5)
          except KeyboardInterrupt:
              my_observer.stop()
              my_observer.join()    
  </code></pre>