title: Scrape all of sublinks of a website recursively in Python using Beautiful Soup
body: <p>NEWEST UPDATE:
  I'm reducing my question to how to get all links from a site, including sublinks of each page etc, recursively.</p>
  
  <p>I think I know how to get all sublinks of one page:</p>
  
  <pre><code>from bs4 import BeautifulSoup
  import requests
  import re
  
  def get_links(site, filename):
      f=open(filename, 'w')
      url = requests.get(site)
      data = url.text
      soup = BeautifulSoup(data, 'lxml')
      for links in soup.find_all('a'):
          f.write(str(links.get('href'))+"\n")
      f.close()
  
  r="https://en.wikipedia.org/wiki/Main_Page"
  filename="wiki"
  get_links(r,filename)
  </code></pre>
  
  <p>How do I recursively ensure all links on the site are also harvested and written onto the same file? </p>
  
  <p>So I tried this, and it's not even compiling.</p>
  
  <pre><code>def is_url(link):
      #checks using regex if 'link' is a valid url
      url = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&amp;+#]|[!*/\\,() ]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', link)
      return (" ".join(url)==link)
  
  def get_links(site, filename):
      f=open(filename, 'a')
      url = requests.get(site)
      data = url.text
      soup = BeautifulSoup(data, 'lxml')
      for links in soup.find_all('a'):
          if is_url(links):
              f.write(str(links.get('href'))+"\n")
              get_links(links, filename)
      f.close()
  </code></pre>