title: Python - webscraping to go several depth-levels within a page with requests module
body: <p>I have a Python3 script that is performing web scraping based on the urls provided in a csv file.
  I am trying to achieve the following:</p>
  <p>1.) Get a page from the URL provided in the CSV file</p>
  <p>2.) Scrape it and search for email addresses with regex + beautifulsoup, then, if email is found, save it to a results.csv file</p>
  <p>3.) Search for all other  (links) on the page</p>
  <p>4.) Go to/get all the links found in the 1st page (1st level of scraping) and do the same</p>
  <p>5.) Perform the same based on the user's defined level of depth (if the user would say go 3 levels deeper than it would do this: Get the page from 1st level (url from CSV file) and do what is needed on that page -&gt; Get all the pages from 2nd level (scraped links from 1st level) and do what is needed -&gt; Get all the pages from 3rd level (scraped links from 2nd level) and do what is needed -&gt; and so on...</p>
  <p>How do I create a loop which would take care of the depth-level scraping?
  I have tried playing with multiple variants of for and while loops, but I am unable to come up with a working solution.</p>
  <p>This is the code that I currently have (currently it's only able to take care of the 1st level scraping):</p>
  <pre><code>from bs4 import BeautifulSoup
  import requests
  import csv
  import re
  
  import time
  import sys, os
  
  #Type the amount of max level of depth for this instance of script
  while True:
      try:
          max_level_of_depth = int(input('Max level of depth for webscraping (must be a number - integer): '))
          print('Do not open the input and neither the output CSV files before the script finishes!')
          break
      except:
          print('You must type a number (integer)! Try again...\n')
          
  #Read the csv file with urls
  with open('urls.csv', mode='r') as urls:
      #Loop through each url from the csv file
      for url in urls:
          #Strip the url from empty new lines
          url_from_csv_to_scrape = url.rstrip('\n')
          print('[FROM CSV] Going to ' + url_from_csv_to_scrape)
          #time.sleep(3)
          i = 1
          #Get the content of the webpage
          page = requests.get(url_from_csv_to_scrape)
          page_content = page.text
          soup = BeautifulSoup(page_content, 'lxml')
          #Find all &lt;p&gt; tags on the page
          paragraphs_on_page = soup.find_all('p')
          for paragraph in paragraphs_on_page:
              #Search for email address in the 1st level of the page
              emails = re.findall(r'[a-zA-Z0-9_\-\.]+@[a-zA-Z0-9_\-\.]+\.[a-zA-Z]{2,5}', str(paragraph))
              #If some emails are found on the webpage, save them to csv
              if emails:
                  with open('results.csv', mode='a') as results:
                      for email in emails:
                          print(email)
                          if email.endswith(('.jpg', '.jpeg', '.png', '.JPG', '.JPEG', '.PNG')):
                              continue
                          results.write(url_from_csv_to_scrape + ', ' + email + '\n')
                          print('Found an email. Saved it to the output file.\n')
                      results.close()
          #Find all &lt;a&gt; tags on the page
          links_on_page = soup.find_all('a')
          #Initiate a list with all links which will later be populated with all found urls to be crawled
          found_links_with_href = []
          #Loop through all the &lt;a&gt; tags on the page
          for link in links_on_page:
              try:
                  #If &lt;a&gt; tag has href attribute
                  if link['href']:
                      link_with_href = link['href']
                      #If the link from the webpage does not have domain and protocol in it, prepend them to it
                      if re.match(r'https://', link_with_href) is None and re.match(r'http://', link_with_href) is None:
                          #If the link already has a slash in it, remove it because it will be added after prepending
                          link_with_href = re.sub(r'/', '', link_with_href)
                          #Prepend the domain and protocol in front of the link
                          link_with_href = url_from_csv_to_scrape + link_with_href
                          #print(link_with_href)
                      found_links_with_href.append(link_with_href)
                      found_links_with_href_backup = found_links_with_href
              except:
                  #If &lt;a&gt; tag does not have href attribute, continue
                  print('No href attribute found, going to next &lt;a&gt; tag...')
                  continue
  </code></pre>
  <p>Any help is much appreciated.</p>
  <p>Thank you</p>