title: Using Selenium with Chrome driver on Ubuntu server which does not have GUI
body: <p>I want so scrape some dynamic url (the page is built using JavaScript and does a redirect to an external page). I understand that I need to use a headless browser and I am using Selenium with Chrome driver. The following code does what I want on my Windows machine:</p>
  <pre class="lang-py prettyprint-override"><code>from selenium import webdriver
  from webdriver_manager.chrome import ChromeDriverManager
  
  class MySpider(scrapy.Spider):
      name = 'my_spider'
  
      def __init__(self):
          self.driver = webdriver.Chrome(ChromeDriverManager().install())
  
      def parse(self, response):
          link = &quot;some-url-which-uses-javascript.com&quot;
          self.driver.get(link)
          time.sleep(1) # without this wait, driver.current_url is not the final redirect
          url = self.driver.current_url
  </code></pre>
  <p>But when I run the same code on my Ubuntu server (which does not have GUI) I get the following error:</p>
  <blockquote>
  <p>builtins.ValueError: Could not get version for Chrome with this
  command: google-chrome --version || google-chrome-stable --version</p>
  </blockquote>
  <p>I have installed both <strong>google-chrome-stable</strong> and <strong>chromium-chromedriver</strong> on the Ubuntu server.</p>
  <p>I have also tried the following code:</p>
  <pre class="lang-py prettyprint-override"><code>class MySpider(scrapy.Spider):
      name = 'my_spider'
  
      def __init__(self):
          # self.driver = webdriver.Chrome(ChromeDriverManager().install())
          # self.driver = webdriver.Chrome('/usr/bin/chromedriver')
          chrome_options = Options()
          chrome_options.add_argument('--headless')
          chrome_options.add_argument('--no-sandbox')
          chrome_options.add_argument('--disable-dev-shm-usage')
          self.driver = webdriver.Chrome('/usr/bin/chromedriver', chrome_options=chrome_options)
  </code></pre>
  <p>Bu I get the following error:</p>
  <blockquote>
  <p>selenium.common.exceptions.WebDriverException: Message: unknown error:
  Chrome failed to start: exited abnormally.   (unknown error:
  DevToolsActivePort file doesn't exist)   (The process started from
  chrome location /usr/bin/google-chrome is no longer running, so
  ChromeDriver is assuming that Chrome has crashed.)</p>
  </blockquote>