title: "Failed to decode response from marionette" message in Python/Firefox headless scraping script
body: <p>Good Day, I've done a number of searches on here and google and yet to find a solution that address this problem.</p>
  
  <p>The scenario is: </p>
  
  <p>I have a Python script (2.7) that loops through an number of URLs (e.g. think Amazon pages, scraping reviews).  Each page has the same HTML layout, just scraping different information.  I use Selenium with a headless browser as these pages have javascript that needs to execute to grab the information.</p>
  
  <p>I run this script on my local machine (OSX 10.10).  Firefox is the latest v59.  Selenium is version at 3.11.0 and using geckodriver v0.20.  </p>
  
  <p>This script locally has no issues, it can run through all the URLs and scrape the pages with no issue.</p>
  
  <p>Now when I put the script on my server, the only difference is it is Ubuntu 16.04 (32 bit).  I use the appropriate geckodriver (still v0.20) but everything else is the same (Python 2.7, Selenium 3.11).  It appears to randomly crash the headless browser and then all of the <code>browserObjt.get('url...')</code> no longer work.  </p>
  
  <p>The error messages say:</p>
  
  <blockquote>
    <p>Message: failed to decode response from marionette</p>
  </blockquote>
  
  <p>Any further selenium requests for pages return the error:</p>
  
  <blockquote>
    <p>Message: tried to run command without establishing a connection</p>
  </blockquote>
  
  <hr>
  
  <p>To show some code:</p>
  
  <p>When I create the driver:</p>
  
  <pre><code>    options = Options()
      options.set_headless(headless=True)
  
      driver = webdriver.Firefox(
          firefox_options=options,
          executable_path=config.GECKODRIVER
      )
  </code></pre>
  
  <p><code>driver</code> is passed to the script's function as a parameter <code>browserObj</code> which is then used to call specific pages and then once that loads it is passed to BeautifulSoup for parsing:</p>
  
  <pre><code>browserObj.get(url)
  
  soup = BeautifulSoup(browserObj.page_source, 'lxml')
  </code></pre>
  
  <hr>
  
  <p>The error might be pointing to the BeautifulSoup line which is crashing the browser.</p>
  
  <p>What is likely causing this, and what can I do to resolve the issue?</p>
  
  <hr>
  
  <p>Edit: Adding stack trace which points to the same thing:</p>
  
  <pre><code>Traceback (most recent call last):
    File "main.py", line 164, in &lt;module&gt;
      getLeague
    File "/home/ps/dataparsing/XXX/yyy.py", line 48, in BBB
      soup = BeautifulSoup(browserObj.page_source, 'lxml')
    File "/home/ps/AAA/projenv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 670, in page_source
      return self.execute(Command.GET_PAGE_SOURCE)['value']
    File "/home/ps/AAA/projenv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
      self.error_handler.check_response(response)
    File "/home/ps/AAA/projenv/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
      raise exception_class(message, screen, stacktrace)
  WebDriverException: Message: Failed to decode response from marionette
  </code></pre>
  
  <hr>
  
  <p>Note: This script used to work with Chrome.  Because the server is a 32bit server, I can only use chromedriver v0.33, which only supports Chrome v60-62.  Currently Chrome is v65 and on DigitalOcean I don't seem to have an easy way to revert back to an old version - which is why I am stuck with Firefox.</p>