title: Running a Selenium webscraper in AWS Lambda using Docker
body: <p>I am trying to create a simple python Lambda app using SAM CLI that gets the number of followers for a particular handle. Have looked at ALL tutorials and blog posts and yet have not been able to make it work.</p>
  <p>The build and local test works fine using <code>sam build</code> and <code>sam local invoke</code>, however, after deployment to AWS Lambda it throws the following error.</p>
  <p>Any ideas how to solve this?</p>
  <pre><code>{
    &quot;errorMessage&quot;: &quot;Message: unknown error: Chrome failed to start: crashed.\n  (chrome not reachable)\n  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n&quot;,
    &quot;errorType&quot;: &quot;WebDriverException&quot;,
    &quot;stackTrace&quot;: [
      &quot;  File \&quot;/var/task/app.py\&quot;, line 112, in main\n    data = Twitter(StockInfo_list)\n&quot;,
      &quot;  File \&quot;/var/task/app.py\&quot;, line 36, in Twitter\n    driver = webdriver.Chrome(executable_path=chromedriver_path, options=chrome_optionsdata)\n&quot;,
      &quot;  File \&quot;/var/task/selenium/webdriver/chrome/webdriver.py\&quot;, line 76, in __init__\n    RemoteWebDriver.__init__(\n&quot;,
      &quot;  File \&quot;/var/task/selenium/webdriver/remote/webdriver.py\&quot;, line 157, in __init__\n    self.start_session(capabilities, browser_profile)\n&quot;,
      &quot;  File \&quot;/var/task/selenium/webdriver/remote/webdriver.py\&quot;, line 252, in start_session\n    response = self.execute(Command.NEW_SESSION, parameters)\n&quot;,
      &quot;  File \&quot;/var/task/selenium/webdriver/remote/webdriver.py\&quot;, line 321, in execute\n    self.error_handler.check_response(response)\n&quot;,
      &quot;  File \&quot;/var/task/selenium/webdriver/remote/errorhandler.py\&quot;, line 242, in check_response\n    raise exception_class(message, screen, stacktrace)\n&quot;
    ]
  }
  </code></pre>
  <p>I'm using the following as my Dockerfile</p>
  <pre><code>FROM public.ecr.aws/lambda/python:3.8
  
  # Update repository and install unzip
  RUN yum update -y
  RUN yum install unzip -y
  
  # Download and install Google Chrome
  COPY curl https://intoli.com/install-google-chrome.sh | bash
  
  # Download and install ChromeDriver
  RUN CHROME_DRIVER_VERSION=`curl -sS https://chromedriver.storage.googleapis.com/LATEST_RELEASE` &amp;&amp; \
      wget -O /tmp/chromedriver.zip https://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip &amp;&amp; \
      unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/
  RUN echo $(chromedriver --version)
  
  # Upgrade PIP
  RUN /var/lang/bin/python3.8 -m pip install --upgrade pip
  
  # Install requirements (including selenium)
  COPY app.py requirements.txt ./
  RUN python3.8 -m pip install -r requirements.txt -t .
  
  # Command can be overwritten by providing a different command in the template directly.
  CMD [&quot;app.main&quot;]
  </code></pre>
  <p>My applications looks like</p>
  <pre class="lang-py prettyprint-override"><code>from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.wait import WebDriverWait
  import os
  import time, datetime
  import pandas as pd
  import csv
  
  
  # import mysql.connector
  from datetime import date, datetime as dt1
  
  
  def Twitter(twitter_stock_id):
  
      twitterlist = []
      stockids = []
      twitterlist = twitter_stock_id[&quot;TwitterUrl&quot;].str.lower().tolist()
      
      stockids = twitter_stock_id[&quot;stockid&quot;].str.lower().tolist()
  
      chrome_optionsdata = webdriver.ChromeOptions()
      prefs = {&quot;profile.default_content_setting_values.notifications&quot;: 2}
      chrome_optionsdata.add_experimental_option(&quot;prefs&quot;, prefs)
      chrome_optionsdata.add_argument(&quot;--headless&quot;)
      chrome_optionsdata.add_argument(&quot;--no-sandbox&quot;)
      chrome_optionsdata.add_argument(&quot;--disable-dev-shm-usage&quot;)
      chrome_optionsdata.add_argument(&quot;--disable-gpu&quot;)
      chrome_optionsdata.add_argument(&quot;--disable-gpu-sandbox&quot;)
  
      chromedriver_path = &quot;/usr/local/bin/chromedriver&quot;
      driver = webdriver.Chrome(executable_path=chromedriver_path, options=chrome_optionsdata)
  
      lfollowers = []
  
      for url in twitterlist:
  
          tempurl = driver.get(url)
          time.sleep(10)
          followers = driver.find_elements_by_class_name(&quot;r-qvutc0&quot;)
          for fol in followers:
              if &quot;Followers&quot; in fol.text:
                  tempstr = fol.text.split(&quot; &quot;)
                  lfollowers.append(tempstr[0])
  
          time.sleep(5)
  
      
      lmaindata = []
      for fld in lfollowers:
          if fld != &quot;Followers&quot;:
              lmaindata.append(fld)
  
      print(&quot;Followers&quot; + str(lmaindata))
      driver.quit()
      return f&quot;Followers: {lmaindata}&quot;
  
  
  import json
  
  
  def main(event, context):
      StockInfo_list = pd.DataFrame([{&quot;TwitterUrl&quot;: &quot;https://twitter.com/costco&quot;, &quot;stockid&quot;: &quot;COST&quot;}])
      data = Twitter(StockInfo_list)
  
      return {
          &quot;statusCode&quot;: 200,
          &quot;body&quot;: json.dumps({&quot;message&quot;: &quot;hello&quot;, &quot;data&quot;: data}),
      }
  </code></pre>