title: AWS Lambda Container Running Selenium With Headless Chrome Works Locally But Not In AWS Lambda
body: <p>I am currently developing a Python program which has a segment which uses a headless version of Chrome and Selenium to perform a repetitive process. I am aiming to run the program on Lambda.</p>
  <p>The overall program has around 1GB of dependencies so the option to use the standard method of using a <a href="https://docs.aws.amazon.com/lambda/latest/dg/python-package.html" rel="noreferrer">.zip archive</a>, containing all my function code and dependencies is not an option as the total unzipped size of the function and all layers can't exceed the unzipped deployment package size limit of 250 MB.</p>
  <p>So, that is where the new <a href="https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/" rel="noreferrer">AWS Lambda – Container Image Support</a> (I used this linked tutorial to develop this whole implementation so please read if you need more info) comes in. This allows me to package and deploy my Lambda function as container images of up to 10 GB in size.</p>
  <p>I am using the base image hosted in ECR Public provided by AWS which runs Amazon Linux 2. Firstly - in my Dockerfile I:</p>
  <ul>
  <li>Download the base image.</li>
  <li>Define some global variables.</li>
  <li>Copy my files over.</li>
  <li>Install my pip appendices</li>
  <li>Use yum to install some packages.</li>
  </ul>
  <p>and finally - I install both Chrome (87.0.4280.88 at the time of reading) and Chromedriver (87.0.4280.88)</p>
  <ul>
  <li>Finally download install both latest versions of Chrome and Chromedriver</li>
  </ul>
  <p>there is a possibility this could be where the problem lies, but I highly doubt this as both are the same version - <a href="https://www.chromium.org/developers/version-numbers" rel="noreferrer">ChromeDriver uses the same version number scheme as Chrome</a>.</p>
  <p>This is my <strong>Dockerfile</strong>:</p>
  <pre><code># 1) DOWNLOAD BASE IMAGE.
  FROM public.ecr.aws/lambda/python:3.8
  
  # 2) DEFINE GLOBAL ARGS.
  ARG MAIN_FILE=&quot;main.py&quot;
  ARG ENV_FILE=&quot;params.env&quot;
  ARG REQUIREMENTS_FILE=&quot;requirements.txt&quot;
  ARG FUNCTION_ROOT=&quot;.&quot;
  ARG RUNTIME_VERSION=&quot;3.8&quot;
  
  # 3) COPY FILES.
  # Copy The Main .py File.
  COPY ${MAIN_FILE} ${LAMBDA_TASK_ROOT}
  # Copy The .env File.
  COPY ${ENV_FILE} ${LAMBDA_TASK_ROOT}
  # Copy The requirements.txt File.
  COPY ${REQUIREMENTS_FILE} ${LAMBDA_TASK_ROOT}
  # Copy Helpers Folder.
  COPY helpers/ ${LAMBDA_TASK_ROOT}/helpers/
  # Copy Private Folder.
  COPY priv/ ${LAMBDA_TASK_ROOT}/priv/
  # Copy Source Data Folder.
  COPY source_data/ ${LAMBDA_TASK_ROOT}/source_data/
  
  # 4) INSTALL DEPENDENCIES.
  RUN --mount=type=cache,target=/root/.cache/pip python3.8 -m pip install --upgrade pip
  RUN --mount=type=cache,target=/root/.cache/pip python3.8 -m pip install wheel
  RUN --mount=type=cache,target=/root/.cache/pip python3.8 -m pip install urllib3
  RUN --mount=type=cache,target=/root/.cache/pip python3.8 -m pip install -r requirements.txt --default-timeout=100
  
  # 5) DOWNLOAD &amp; INSTALL CHROMEIUM + CHROMEDRIVER.
  #RUN yum -y upgrade
  RUN yum -y install wget unzip libX11 nano wget unzip xorg-x11-xauth xclock xterm
  
  # Install Chrome
  RUN wget https://intoli.com/install-google-chrome.sh
  RUN bash install-google-chrome.sh
  
  # Install Chromedriver
  RUN wget https://chromedriver.storage.googleapis.com/87.0.4280.88/chromedriver_linux64.zip
  RUN unzip ./chromedriver_linux64.zip
  RUN rm ./chromedriver_linux64.zip
  RUN mv -f ./chromedriver /usr/local/bin/chromedriver
  RUN chmod 755 /usr/local/bin/chromedriver
  
  # 5) SET CMD OF HANDLER.
  CMD [ &quot;main.lambda_handler&quot; ]
  </code></pre>
  <p>This image always builds without a problem and creates my image as expected.</p>
  <p>and my <strong>docker-compose.yml</strong> file:</p>
  <pre><code>version: &quot;3.7&quot;
  services:
    lambda:
      image: tbg-lambda:latest
      build: .
      ports:
        - &quot;8080:8080&quot;
      env_file:
       - ./params.env
  </code></pre>
  <p>So - now that the image is build I can test locally with cURL. Here, I am passing an empty JSON payload:</p>
  <pre><code>curl -XPOST &quot;http://localhost:8080/2015-03-31/functions/function/invocations&quot; -d '{}'
  </code></pre>
  <p>which runs the whole program perfectly start to end using Chrome headless mode with no errors.</p>
  <p>So great - the Docker container works locally and as expected.</p>
  <p>Lets upload it to ECR so I can use it with my Lambda Function (ECR URL changed for security):</p>
  <pre><code>aws ecr create-repository --repository-name tbg-lambda:latest --image-scanning-configuration scanOnPush=true
  docker tag tbg-lambda:latest 123412341234.dkr.ecr.sa-east-1.amazonaws.com/tbg-lambda:latest
  aws ecr get-login-password | docker login --username AWS --password-stdin 123412341234.dkr.ecr.sa-east-1.amazonaws.com
  docker push 123412341234.dkr.ecr.sa-east-1.amazonaws.com/tbg-lambda:latest 
  </code></pre>
  <p>Everything pushes up as expected - I then create my new lambda function, choosing &quot;Container Image&quot; as the function option and attach the IAM role with all the permissions I need:</p>
  <p><a href="https://i.stack.imgur.com/pNZnU.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pNZnU.png" alt="IAM Role" /></a></p>
  <p>I have the memory set the max value just to ensure this isn't the problem:</p>
  <p><a href="https://i.stack.imgur.com/1pUO5.png" rel="noreferrer"><img src="https://i.stack.imgur.com/1pUO5.png" alt="enter image description here" /></a></p>
  <p>Ok - so lets to get to the <strong>point of failure:</strong></p>
  <p>I use a test event to invoke the function through the console:
  <a href="https://i.stack.imgur.com/kKHzm.png" rel="noreferrer"><img src="https://i.stack.imgur.com/kKHzm.png" alt="enter image description here" /></a></p>
  <p>Everything runs perfectly until it hits the code which creates the webdriver driver with Chrome:</p>
  <pre><code>    options = Options()
      options.add_argument('--no-sandbox')
      options.add_argument('--headless')
      options.add_argument('--single-process')
      options.add_argument('--disable-dev-shm-usage')
      options.add_argument('--remote-debugging-port=9222')
      options.add_argument('--disable-infobars')
      driver = webdriver.Chrome(
          service_args=[&quot;--verbose&quot;, &quot;--log-path={}&quot;.format(logPath)],
          executable_path=f&quot;/usr/local/bin/chromedriver&quot;,
          options=options
      )
  </code></pre>
  <p>PS: logPath is just another folder in the project directory - the logs output here as expected, the logs are shown below.</p>
  <p><strong>Heres is the part of the Cloudwatch Logs where the error is highlighted:</strong></p>
  <pre><code>Caught WebDriverException Error: unknown error: Chrome failed to start: crashed.
  
  (unknown error: DevToolsActivePort file doesn't exist)
  
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  
  END RequestId: 7c933bca-5f0d-4458-9529-db28da677444
  
  REPORT RequestId: 7c933bca-5f0d-4458-9529-db28da677444 Duration: 59104.94 ms Billed Duration: 59105 ms Memory Size: 10240 MB Max Memory Used: 481 MB
  
  RequestId: 7c933bca-5f0d-4458-9529-db28da677444 Error: Runtime exited with error: exit status 1 Runtime.ExitError 
  </code></pre>
  <p><strong>And here is the full Chromedriver log file:</strong></p>
  <pre><code>[1608748453.064][INFO]: Starting ChromeDriver 87.0.4280.88 (89e2380a3e36c3464b5dd1302349b1382549290d-refs/branch-heads/4280@{#1761}) on port 54581
  [1608748453.064][INFO]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
  [1608748453.064][INFO]: /dev/shm not writable, adding --disable-dev-shm-usage switch
  [1608748453.679][SEVERE]: CreatePlatformSocket() failed: Address family not supported by protocol (97)
  [1608748453.679][INFO]: listen on IPv6 failed with error ERR_ADDRESS_UNREACHABLE
  [1608748454.432][INFO]: [13826d22c628514ca452d1f2949eb011] COMMAND InitSession {
     &quot;capabilities&quot;: {
        &quot;alwaysMatch&quot;: {
           &quot;browserName&quot;: &quot;chrome&quot;,
           &quot;goog:chromeOptions&quot;: {
              &quot;args&quot;: [ &quot;--no-sandbox&quot;, &quot;--headless&quot;, &quot;--single-process&quot;, &quot;--disable-dev-shm-usage&quot; ],
              &quot;extensions&quot;: [  ]
           },
           &quot;platformName&quot;: &quot;any&quot;
        },
        &quot;firstMatch&quot;: [ {
  
        } ]
     },
     &quot;desiredCapabilities&quot;: {
        &quot;browserName&quot;: &quot;chrome&quot;,
        &quot;goog:chromeOptions&quot;: {
           &quot;args&quot;: [ &quot;--no-sandbox&quot;, &quot;--headless&quot;, &quot;--single-process&quot;, &quot;--disable-dev-shm-usage&quot; ],
           &quot;extensions&quot;: [  ]
        },
        &quot;platform&quot;: &quot;ANY&quot;,
        &quot;version&quot;: &quot;&quot;
     }
  }
  [1608748454.433][INFO]: Populating Preferences file: {
     &quot;alternate_error_pages&quot;: {
        &quot;enabled&quot;: false
     },
     &quot;autofill&quot;: {
        &quot;enabled&quot;: false
     },
     &quot;browser&quot;: {
        &quot;check_default_browser&quot;: false
     },
     &quot;distribution&quot;: {
        &quot;import_bookmarks&quot;: false,
        &quot;import_history&quot;: false,
        &quot;import_search_engine&quot;: false,
        &quot;make_chrome_default_for_user&quot;: false,
        &quot;skip_first_run_ui&quot;: true
     },
     &quot;dns_prefetching&quot;: {
        &quot;enabled&quot;: false
     },
     &quot;profile&quot;: {
        &quot;content_settings&quot;: {
           &quot;pattern_pairs&quot;: {
              &quot;https://*,*&quot;: {
                 &quot;media-stream&quot;: {
                    &quot;audio&quot;: &quot;Default&quot;,
                    &quot;video&quot;: &quot;Default&quot;
                 }
              }
           }
        },
        &quot;default_content_setting_values&quot;: {
           &quot;geolocation&quot;: 1
        },
        &quot;default_content_settings&quot;: {
           &quot;geolocation&quot;: 1,
           &quot;mouselock&quot;: 1,
           &quot;notifications&quot;: 1,
           &quot;popups&quot;: 1,
           &quot;ppapi-broker&quot;: 1
        },
        &quot;password_manager_enabled&quot;: false
     },
     &quot;safebrowsing&quot;: {
        &quot;enabled&quot;: false
     },
     &quot;search&quot;: {
        &quot;suggest_enabled&quot;: false
     },
     &quot;translate&quot;: {
        &quot;enabled&quot;: false
     }
  }
  [1608748454.433][INFO]: Populating Local State file: {
     &quot;background_mode&quot;: {
        &quot;enabled&quot;: false
     },
     &quot;ssl&quot;: {
        &quot;rev_checking&quot;: {
           &quot;enabled&quot;: false
        }
     }
  }
  [1608748454.433][INFO]: Launching chrome: /usr/bin/google-chrome --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --headless --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=0 --single-process --test-type=webdriver --use-mock-keychain --user-data-dir=/tmp/.com.google.Chrome.xgjs0h data:,
  mkdir: cannot create directory ‘/.local’: Read-only file system
  touch: cannot touch ‘/.local/share/applications/mimeapps.list’: No such file or directory
  /usr/bin/google-chrome: line 45: /dev/fd/62: No such file or directory
  /usr/bin/google-chrome: line 46: /dev/fd/62: No such file or directory
  prctl(PR_SET_NO_NEW_PRIVS) failed
  [1223/183429.578846:FATAL:zygote_communication_linux.cc(255)] Cannot communicate with zygote
  Failed to generate minidump.[1608748469.769][INFO]: [13826d22c628514ca452d1f2949eb011] RESPONSE InitSession ERROR unknown error: Chrome failed to start: crashed.
    (unknown error: DevToolsActivePort file doesn't exist)
    (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  [1608748469.769][DEBUG]: Log type 'driver' lost 0 entries on destruction
  [1608748469.769][DEBUG]: Log type 'browser' lost 0 entries on destruction
  </code></pre>
  <p>One thing that I might think could be the problem would the way lambda is running this container vs how I am running it locally.</p>
  <p>Alot of people reccomend <strong>NOT</strong> to not run chrome as root - so is Lambda running the container as root and thats what is causing this? If so how can I tell Lambda or Docker to run the code as a non-root user.</p>
  <p>This is mentioned here: <a href="https://github.com/heroku/heroku-buildpack-google-chrome/issues/46#issuecomment-484562558" rel="noreferrer">https://github.com/heroku/heroku-buildpack-google-chrome/issues/46#issuecomment-484562558</a></p>
  <p>I have been fighting with this error pretty much since AWS announced the lambda containers so any help with this would be brilliant  Please ask for any more info if I missed something!</p>
  <p>Thanks in advance.</p>