title: How can I make sure a BS4 request is being made with a socket on a list?
body: <p>I have a list of proxies like this one I would like to use in scraping with python:</p>
  
  <pre><code>proxies_ls = [  '149.56.89.166:3128',
              '194.44.176.116:8080',
              '14.203.99.67:8080',
              '185.87.65.204:63909',
              '103.206.161.234:63909',
              '110.78.177.100:65103']
  </code></pre>
  
  <p>and made a function in order to scrap a url using bs4 and requests module called crawlSite(url). Here's the code:</p>
  
  <pre><code># Bibliotecas para crawl e regex
  from bs4 import BeautifulSoup
  import requests
  from fake_useragent import UserAgent
  import re
  
  #Biblioteca para data
  import datetime
  from time import gmtime, strftime
  
  #Biblioteca para escrita dos logs
  import os
  import errno
  
  #Biblioteca para delay aleatorio
  import time
  import random
  
  print('BOT iniciado: '+ datetime.datetime.now().strftime('%d-%m-%Y %H:%M:%S'))
  
  proxies_ls = [  '149.56.89.166:3128',
              '194.44.176.116:8080',
              '14.203.99.67:8080',
              '185.87.65.204:63909',
              '103.206.161.234:63909',
              '110.78.177.100:65103']
  
  def crawlSite(url):
      #Chrome emulation
      ua=UserAgent()
      header={'user-agent':ua.chrome}
      random.shuffle(proxies_ls)
  
      #Random delay
      print('antes do delay: '+ datetime.datetime.now().strftime('%d-%m-%Y %H:%M:%S'))
      tempoRandom=random.randint(1,5)
      time.sleep(tempoRandom)
  
      try:
          randProxy=random.choice(proxies_ls)
          # Getting the webpage, creating a Response object emulated with chrome with a 30sec timeout.
          response = requests.get(url,proxies = {'https':randProxy},headers=header,timeout=30)
          print(response)
          print('Resposta obtida: '+ datetime.datetime.now().strftime('%d-%m-%Y %H:%M:%S'))
  
          #Avoid HTTP request errors
          if response.status_code == 404:
              raise ConnectionError("HTTP Response [404] - The requested resource could not be found")
          elif response.status_code == 409:            
              raise ConnectionError("HTTP Response [409] - Possible Cloudflare DNS resolution error")
          elif response.status_code == 403:
              raise ConnectionError("HTTP Response [403] - Permission denied error")
          elif response.status_code == 503:
              raise ConnectionError("HTTP Response [503] - Service unavailable error")
          print('RR Status {}'.format(response.status_code))
          # Extracting the source code of the page.
          data = response.text
  
      except ConnectionError:
          try:
              proxies_ls.remove(randProxy)
          except ValueError:
              pass
          randProxy=random.choice(proxies_ls)
  
      return BeautifulSoup(data, 'lxml')
  </code></pre>
  
  <p>What I would like to do is to make sure only the proxies on that list are being used in the conection. 
  The random part </p>
  
  <pre><code> randProxy=random.choice(proxies_ls)
  </code></pre>
  
  <p>is working ok but the checking part if the proxy is valid or not isn't. Mainly because I still get 200 as response with a "made up proxy".</p>
  
  <p>If I reduce the list to this:</p>
  
  <pre><code>proxies_ls = ['149.56.89.166:3128']
  </code></pre>
  
  <p>with a proxy that doesn't work I still get 200 as response! (I tried using a proxychecker like <a href="https://pt.infobyip.com/proxychecker.php" rel="nofollow noreferrer">https://pt.infobyip.com/proxychecker.php</a> and it doesn't work...)</p>
  
  <p>So my questions are (I'll enumerate so it is easier):
  a) Why am I getting this 200 response and not a 4xx response?
  b) How can I force the request to use the proxies as I want?</p>
  
  <p>Thank you,</p>
  
  <p>Eunito.</p>