title: Web Scraper freezes on digital ocean vps
body: <p>I have this scraper build with asyncio and httpx and it triggers on POST request where a user uploads the list of keywords as a csv file. Everything working fine locally but it hangs up when I try to do 50+ keywords on digital ocean server.</p>
  <p>my api code fragment:</p>
  <pre><code>import fastapi as _fastapi
  from fastapi.responses import HTMLResponse, FileResponse, Response, RedirectResponse, StreamingResponse
  from starlette.requests import Request
  from starlette.templating import Jinja2Templates
  import shutil
  import os
  import time
  
  from rq import Queue
  from rq.job import Job
  
  from redis import Redis
  
  from scraper import run_scraper
  from utils import clean_file, csv_writer
  
  app = _fastapi.FastAPI()
  
  r = Redis(
      host=&quot;localhost&quot;,
      port=6379,
      db=0,
  )
  q = Queue(connection=r, default_timeout=-1)
  
  templates = Jinja2Templates(&quot;templates&quot;)
  
  
  
  
  @app.get(&quot;/&quot;)
  def index(request: Request):
      return templates.TemplateResponse(&quot;index.html&quot;, {&quot;request&quot;: request})
  
  
  @app.post(&quot;/api/v1/scraped_csv&quot;)
  async def extract_ads(csv_file: _fastapi.UploadFile = _fastapi.File(...)):
      temp_file = _save_file_to_disk(csv_file, path=&quot;temp&quot;, save_as=&quot;temp&quot;)
      task = q.enqueue(run_scraper, temp_file)
  
      return RedirectResponse(f'/progress/{task.id}', status_code=303)
      
  
  
  @app.get(&quot;/progress/{job_id}&quot;)
  def progress(request: Request, job_id):
      job = Job.fetch(job_id, connection=r)
      if job.is_finished:
          csv_path = os.path.abspath(clean_file)
          return FileResponse(path=csv_path, media_type=&quot;text/csv&quot;, filename=clean_file)
      return templates.TemplateResponse(&quot;log_stream.html&quot;, {&quot;request&quot;: request})
  
  
  @app.get(&quot;/log_stream/&quot;)
  def stream():
      def iterfile():
          with open(&quot;scraper.log&quot;) as log_info:
              while True:
                  where = log_info.tell()
                  line = log_info.readline()
                  if not line:
                      time.sleep(0.5)
                      log_info.seek(where)
                  else:
                      yield line
  
     
      return StreamingResponse(iterfile(), media_type=&quot;text/plain&quot;)
  
  
  def _save_file_to_disk(uploaded_file, path=&quot;.&quot;, save_as=&quot;default&quot;):
      extension = os.path.splitext(uploaded_file.filename)[-1]
      temp_file = os.path.join(path, save_as + extension)
      with open(temp_file, &quot;wb&quot;) as buffer:
          shutil.copyfileobj(uploaded_file.file, buffer)
      return temp_file
  </code></pre>
  <p>I am using redis queue and make scraping in the background because of request timeout issue.</p>
  <p>my scraper code fragment:</p>
  <pre><code>import httpx
  import asyncio
  from bs4 import BeautifulSoup
  from decouple import config
  from urllib.parse import urlencode
  from urllib.parse import urlparse
  import os
  import logging
  
  from utils import csv_reader, csv_writer
  
  SCRAPERAPI_KEY = config(&quot;API_KEY&quot;)
  NUM_RETRIES = 3
  
  logging.basicConfig(filename=&quot;scraper.log&quot;,
                      format='%(asctime)s %(message)s',
                      filemode='w')
  
  logger = logging.getLogger()
  logger.setLevel(logging.DEBUG)
  
  
  base_url = &quot;https://www.google.com/search?&quot;
  
  headers = {
      &quot;authority&quot;: &quot;www.google.com&quot;,
      &quot;sec-ch-dpr&quot;: &quot;1&quot;,
      &quot;sec-ch-viewport-width&quot;: &quot;1366&quot;,
      &quot;sec-ch-ua&quot;: '&quot; Not A;Brand&quot;;v=&quot;99&quot;, &quot;Chromium&quot;;v=&quot;98&quot;, &quot;Yandex&quot;;v=&quot;22&quot;',
      &quot;sec-ch-ua-mobile&quot;: &quot;?0&quot;,
      &quot;sec-ch-ua-full-version&quot;: '&quot;22.3.3.886&quot;',
      &quot;sec-ch-ua-arch&quot;: '&quot;x86&quot;',
      &quot;sec-ch-ua-platform&quot;: '&quot;Linux&quot;',
      &quot;sec-ch-ua-platform-version&quot;: '&quot;5.4.0&quot;',
      &quot;sec-ch-ua-model&quot;: '&quot;&quot;',
      &quot;sec-ch-ua-bitness&quot;: '&quot;64&quot;',
      &quot;sec-ch-ua-full-version-list&quot;: '&quot; Not A;Brand&quot;;v=&quot;99.0.0.0&quot;, &quot;Chromium&quot;;v=&quot;98.0.4758.886&quot;, &quot;Yandex&quot;;v=&quot;22.3.3.886&quot;',
      &quot;upgrade-insecure-requests&quot;: &quot;1&quot;,
      &quot;user-agent&quot;: &quot;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.141 Safari/537.36&quot;,
      &quot;accept&quot;: &quot;text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9&quot;,
      &quot;sec-fetch-site&quot;: &quot;same-origin&quot;,
      &quot;sec-fetch-mode&quot;: &quot;navigate&quot;,
      &quot;sec-fetch-user&quot;: &quot;?1&quot;,
      &quot;sec-fetch-dest&quot;: &quot;document&quot;,
      &quot;referer&quot;: &quot;https://www.google.com/&quot;,
      &quot;accept-language&quot;: &quot;en,ru;q=0.9&quot;,
  }
  
  pagination_params = {
      &quot;q&quot;: &quot;lift installation&quot;,
      &quot;source&quot;: &quot;lnms&quot;,
      &quot;tbm&quot;: &quot;shop&quot;,
      &quot;ei&quot;: &quot;aW-HYv74MrCOseMP_8OumAY&quot;,
      &quot;start&quot;: &quot;&quot;,
      &quot;sa&quot;: &quot;X&quot;,
      &quot;ved&quot;: &quot;0ahUKEwikobSm8e33AhUjUGwGHQVfDr84PBDy0wMIpw0&quot;,
      &quot;biw&quot;: &quot;480&quot;,
      &quot;bih&quot;: &quot;665&quot;,
      &quot;dpr&quot;: &quot;1&quot;,
  }
  
  initial_params = {
      &quot;q&quot;: &quot;&quot;,
      &quot;source&quot;: &quot;lnms&quot;,
      &quot;tbm&quot;: &quot;shop&quot;,
      &quot;sa&quot;: &quot;X&quot;,
      &quot;ved&quot;: &quot;0ahUKEwikobSm8e33AhUjUGwGHQVfDr84PBDy0wMIpw0&quot;,
      &quot;biw&quot;: &quot;480&quot;,
      &quot;bih&quot;: &quot;665&quot;,
      &quot;dpr&quot;: &quot;1&quot;,
  }
  
  results = []
  
  
  def get_scraperapi_url(url):
      payload = {
          &quot;api_key&quot;: SCRAPERAPI_KEY,
          &quot;url&quot;: url,
          &quot;country_code&quot;: &quot;au&quot;,
          &quot;keep_headers&quot;: &quot;true&quot;,
      }
      proxy_url = &quot;http://api.scraperapi.com/?&quot; + urlencode(payload)
      return proxy_url
  
  
  async def log_request(request):
      logger.debug(f&quot;Request: {request.method} {request.url}&quot;)
  
  
  async def log_response(response):
      request = response.request
      logger.debug(f&quot;Response: {request.method} {request.url} - Status: {response.status_code}&quot;)
  
  
  async def fetch_pages(keyword, page_no):
      initial_params[&quot;q&quot;] = keyword
      if not page_no:
          params = initial_params
          url = base_url + urlencode(params)
      else:
          params = pagination_params
  
          params[&quot;start&quot;] = str(page_no * 10)
  
          params[&quot;q&quot;] = keyword
          url = base_url + urlencode(params)
  
      async with httpx.AsyncClient(
          event_hooks={&quot;request&quot;: [log_request], &quot;response&quot;: [log_response]}
      ) as client:
          response = await client.get(
              get_scraperapi_url(url), headers=headers, timeout=None
          )
  
          return response
  
  
  async def parse_page(html):
  
      ad_urls = []
      content = BeautifulSoup(html, &quot;lxml&quot;)
  
      for ad in content.find_all(&quot;a&quot;, {&quot;class&quot;: &quot;sh-np__click-target&quot;}):
          try:
              async with httpx.AsyncClient() as client:
                  r = await client.get(
                      &quot;https://www.google.com&quot; + ad[&quot;href&quot;], headers=headers, timeout=None
                  )
                  url = str(r.url)
                  ad_urls.append(urlparse(url).netloc)
                  logger.debug(urlparse(url).netloc)
          except:
              pass
  
      for idx in range(len(ad_urls)):
  
          results.append({&quot;Ad_Url&quot;: ad_urls[idx]})
  
      return results
  
  
  async def run_scraper(file_path):
      tasks = []
      kw = await csv_reader(file_path)
      for k in kw:
          for page in range(0, 4):
              for _ in range(NUM_RETRIES):
                  try:
                      response = await fetch_pages(k, page)
                      if response.status_code in [200, 404]:
                          break
                  except httpx.ConnectError:
                      response = &quot;&quot;
              if response.status_code == 200:
                  tasks.append(asyncio.create_task(parse_page(response.content)))
  
      ad_data = await asyncio.gather(*tasks)
  
      logger.info('Done!')
      await csv_writer(ad_data[0])
      logger.info('csv created.. Please refresh the page to download the csv.')
      
  
      return ad_data[0]
  </code></pre>
  <p>gunicorn.py</p>
  <pre><code>import multiprocessing
  import os
  from dotenv import load_dotenv
  load_dotenv()
  
  name = &quot;gunicorn config for FastAPI - TutLinks.com&quot;
  accesslog = &quot;/root/fastapi_google_ads_scraper/google_ad_scraper/gunicorn-access.log&quot;
  errorlog = &quot;/root/fastapi_google_ads_scraper/google_ad_scraper/gunicorn-error.log&quot;
  
  bind = &quot;0.0.0.0:8000&quot;
  
  worker_class = &quot;uvicorn.workers.UvicornWorker&quot;
  workers = multiprocessing.cpu_count () * 2 + 1
  worker_connections = 1024
  backlog = 2048
  max_requests = 5120
  timeout = 0
  keepalive = 2
  
  debug = os.environ.get(&quot;debug&quot;, &quot;false&quot;) == &quot;true&quot;
  reload = debug
  preload_app = False
  daemon = False
  </code></pre>
  <p>Digital Ocean VPS: 8gb ram, 4 AMD vcpus, 25gb disk, Ubuntu 20.04</p>
  <p>Scraper just hangs up without any errors and I have declared the <code>default_timeout=-1</code> in redis queue because some of the requests do take time more than usual. I have tried to run the scraper independently under vps to see whether it is scraper issue or api but the scraper itself hang up for hours at the last keyword. Any ideas on how to solve this issue? I have been hitting my head against the wall for weeks now but couldn't figure out. Thank you</p>