title: Threading still takes a very long time
body: <p>I have made a script which constructs a checkout URL for shopify websites. This is done by appending each unique product 'variant' ID in the checkout URL and then opening the said URL in a webbrowser. To find the variant ID, i need to parse the website's sitemap to obtain the ID, which I am currenly doing in seperate threads for each product i am parsing, however with each thread added the time it takes increases by quite a lot (nearly one second). </p>
  
  <p>Why is this the case? Shouldn't it take around the same time since each thread basically does the same exact thing? </p>
  
  <p>For reference, one thread takes around 2.0s, two threads 2.8s and three threads around 3.8s</p>
  
  <p>Here is my code:</p>
  
  <pre><code>import time
  import requests
  from bs4 import BeautifulSoup
  import webbrowser
  import threading
  
  sitemap2 = 'https://deadstock.ca/sitemap_products_1.xml'
  atc_url = 'https://deadstock.ca/cart/'
  
  # CHANGE SITEMAP TO THE CORRECT ONE (THE SITE YOU ARE SCRAPING)
  
  variant_list = []
  
  
  def add_to_cart(keywords, size):
      init = time.time()
      # Initialize session
      product_url = ''
      parse_session = requests.Session()
      response = parse_session.get(sitemap2)
      soup = BeautifulSoup(response.content, 'lxml')
      variant_id = 0
  
      # Find Item
      for urls in soup.find_all('url'):
          for images in urls.find_all('image:image'):
              if all(i in images.find('image:title').text.lower() for i in keywords):
                  now = time.time()
                  product_name = images.find('image:title').text
                  print('FOUND: ' + product_name + ' - ' + str(format(now-init, '.3g')) + 's')
                  product_url = urls.find("loc").text
  
      if product_url != '':
          response1 = parse_session.get(product_url+".xml")
          soup = BeautifulSoup(response1.content,'lxml')
          for variants in soup.find_all('variant'):
              if size in variants.find('title').text.lower():
                  variant_id = variants.find('id', type='integer').text
                  atc_link = str(variant_id)+':1'
                  print(atc_link)
                  variant_list.append(atc_link)
  
  
      try:
          print("PARSED PRODUCT: " + product_name)
  
      except UnboundLocalError:
          print("Retrying")
          add_to_cart(keywords, size)
  
  
  def open_checkout():
      url = 'https://deadstock.ca/cart/'
      for var in variant_list:
          url = url + var + ','
      webbrowser.open_new_tab(url)
  
  
  
  
  # When initializing a new thread, only change the keywords in the args, and make sure you start and join the thread.
  # Change sitemap in scraper.py to your websites' sitemap
  
  # If the script finds multiple items, the first item will be opened so please try to be very specific yet accurate.
  
  def main():
      print("Starting Script")
      init = time.time()
  
      try:
          t1 = threading.Thread(target=add_to_cart, args=(['alltimers','relations','t-shirt','white'],'s',))
          t2 = threading.Thread(target=add_to_cart, args=(['alltimers', 'relations', 'maroon'],'s',))
          t3 = threading.Thread(target=add_to_cart, args=(['brain', 'dead','melter'], 's',))
          t1.start()
          t2.start()
          t3.start()
          t1.join()
          t2.join()
          t3.join()
          print(variant_list)
          open_checkout()
      except:
          print("Product not found / not yet live. Retrying..")
          main()
  
      print("Time taken: " + str(time.time()-init))
  
  if __name__ == '__main__':
      main()
  </code></pre>