title: BS4 MemoryError: stack overflow and EOFError: Ran out of input when using multiprocessing in python
body: <p>I have a simple python script that utilizes Python's BS4 library and multiprocessing to do some web scraping. I was initially getting some errors where the script would not complete since I would exceed the recursion limit, but then I found out <a href="https://stackoverflow.com/questions/52021254/maximum-recursion-depth-exceeded-multiprocessing-and-bs4">here</a> that BeautifulSoup trees cannot be pickled, and so causes issues with multiprocessing, so I followed one recommendation in the top answer which was to do the following: <code>sys.setrecursionlimit(25000)</code></p>
  <p>This worked fine for a couple of weeks with no issues (as far as I could tell), but today I restarted the script and some of the processes do not work and I get the error that you can see below:</p>
  <p>I now get this error:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;C:/Users/user/PycharmProjects/foo/single_items/single_item.py&quot;, line 243, in &lt;module&gt;
      Process(target=instance.constant_thread).start()
    File &quot;C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py&quot;, line 112, in start
      self._popen = self._Popen(self)
    File &quot;C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py&quot;, line 223, in _Popen
      return _default_context.get_context().Process._Popen(process_obj)
    File &quot;C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py&quot;, line 322, in _Popen
      return Popen(process_obj)
    File &quot;C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\popen_spawn_win32.py&quot;, line 89, in __init__
      reduction.dump(process_obj, to_child)
    File &quot;C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\reduction.py&quot;, line 60, in dump
      ForkingPickler(file, protocol).dump(obj)
    File &quot;C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\bs4\element.py&quot;, line 1449, in __getattr__
      &quot;'%s' object has no attribute '%s'&quot; % (self.__class__, tag))
  MemoryError: stack overflow
  Traceback (most recent call last):
    File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;
    File &quot;C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\spawn.py&quot;, line 105, in spawn_main
      exitcode = _main(fd)
    File &quot;C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\spawn.py&quot;, line 115, in _main
      self = reduction.pickle.load(from_parent)
  EOFError: Ran out of input
  </code></pre>
  <p>I am not sure what it means, but here is a pseudocode example of the script I have running:</p>
  <pre><code>class foo:
      def __init__(url):
          self.url = url
  
      def constant_scrape:
          while True:
              rq = make_get_request(self.url)
              soup = BeautifulSoup(rq)
  
  
  
  if __name__ == '__main__':
  
      sys.setrecursionlimit(25000)
  
      url_list = [...]
  
      for url in url_list:
          instance = foo(url)
          Process(target=instance.constant_scrape).start()
  </code></pre>
  <hr />
  <p><strong>update 1:</strong>
  It seems be that it is the same URLs every time that crash even though each url is of (seemingly) the same HTML format as the URLs that do not crash.</p>