title: How to scrape and extract all the subcategories names from all its associated pages for a wikipedia category using python 3.6?
body: <p>I want to scrape all the subcategories and pages under the category header of the Category page: "Category:Computer science". The link for the same is as follows: <a href="http://en.wikipedia.org/wiki/Category:Computer_science" rel="nofollow noreferrer">http://en.wikipedia.org/wiki/Category:Computer_science</a>.</p>
  
  <p>I have got an idea regarding the above mentioned problem, from the following stack overflow answer which is specified in the following link.
  <a href="https://stackoverflow.com/questions/41391168/pythonic-beautifulsoup4-how-to-get-remaining-titles-from-the-next-page-link-of">Pythonic beautifulSoup4 : How to get remaining titles from the next page link of a wikipedia category</a>
  and 
  <a href="https://stackoverflow.com/questions/42495405/how-to-scrape-subcategories-and-pages-in-categories-of-a-category-wikipedia-page">How to scrape Subcategories and pages in categories of a Category wikipedia page using Python</a> </p>
  
  <p>However, the answer do not fully solves the problem. It only scrapes the <code>Pages in category "Computer science"</code>. But, I want to extract all the subcategories names and its associated pages. I want the process should report the results in BFS manner with a depth of 10. Is there exist any way to do this?</p>
  
  <p>I found the following code from <strong><em><a href="https://stackoverflow.com/questions/41391168/pythonic-beautifulsoup4-how-to-get-remaining-titles-from-the-next-page-link-of">this linked post</a></em></strong>:</p>
  
  <pre><code>from pprint import pprint
  from urllib.parse import urljoin
  
  from bs4 import BeautifulSoup
  import requests
  
  
  base_url = 'https://en.wikipedia.org/wiki/Category:Computer science'
  
  
  def get_next_link(soup):
      return soup.find("a", text="next page")
  
  def extract_links(soup):
      return [a['title'] for a in soup.select("#mw-pages li a")]
  
  
  with requests.Session() as session:
      content = session.get(base_url).content
      soup = BeautifulSoup(content, 'lxml')
  
      links = extract_links(soup)
      next_link = get_next_link(soup)
      while next_link is not None:  # while there is a Next Page link
          url = urljoin(base_url, next_link['href'])
          content = session.get(url).content
          soup = BeautifulSoup(content, 'lxml')
  
          links += extract_links(soup)
  
          next_link = get_next_link(soup)
  
  pprint(links)
  </code></pre>