title: How to loop through scraping multiple documents on multiple web pages using BeautifulSoup?
body: <p>I am interested in grabbing text from a webpage of a medical documents for a Natural Language Processing project.  The web page document text that I am scraping was not designed with any semantic markup, it's just a big blob of text with bold headings. After getting some help and starting with the first page I am interested in I've implemented the following code to grab the document text from the web page:</p>
  
  <pre><code>import requests
  import re
  from bs4 import BeautifulSoup, Tag, NavigableString, Comment
  
  url = 'https://www.mtsamples.com/site/pages/sample.asp?Type=24- Gastroenterology&amp;Sample=2332-Abdominal%20Abscess%20I&amp;D'
  res = requests.get(url)
  res.raise_for_status()
  html = res.text
  soup = BeautifulSoup(html, 'html.parser')
  
  title_el = soup.find('h1')
  page_title = title_el.text.strip()
  first_hr = title_el.find_next_sibling('hr')
  
  description_title = title_el.find_next_sibling('b', 
  text=re.compile('description', flags=re.I))
  description_text_parts = []
  for s in description_title.next_siblings:
      if s is first_hr:
          break
      if isinstance(s, Tag):
          description_text_parts.append(s.text.strip())
      elif isinstance(s, NavigableString):
          description_text_parts.append(str(s).strip())
  description_text = '\n'.join(p for p in description_text_parts if p.strip())
  
  # titles are all bold and uppercase
  titles = [b for b in first_hr.find_next_siblings('b') if b.text.strip().isupper()]
  
  docs = []
  for t in titles:
      text_parts = []
      for s in t.next_siblings:
          # go until next title
          if s in titles:
              break
          if isinstance(s, Comment):
              continue
          if isinstance(s, Tag):
              if s.name == 'div':
                  break
              text_parts.append(s.text.strip())
          elif isinstance(s, NavigableString):
              text_parts.append(str(s).strip())
      text = '\n'.join(p for p in text_parts if p.strip())
      docs.append({
          'title': t.text.strip(),
          'text': text
      })
  </code></pre>
  
  <p>This will add my document text as a dictionary separated by <code>title</code> field keys and <code>text</code> values to a list named <code>docs</code>.  At this point the web page that was scraped in the above example would be the only element in the <code>docs</code> list.</p>
  
  <p>I am interested in creating a loop to add all medical document records in the Gastroenterology section from the web page found at <a href="https://www.mtsamples.com/site/pages/browse.asp?type=24-Gastroenterology&amp;page=1" rel="nofollow noreferrer">https://www.mtsamples.com/site/pages/browse.asp?type=24-Gastroenterology&amp;page=1</a>.  There are 23 separate pages each with a number of different medical documents in alphabetical order containing a total of 230 medical documents.  I am wondering what the best way to perform this loop would be?  Again, my goal would be to append each medical document to the <code>docs</code> list as shown for the first example in my previous code.  Any help would be much appreciated!</p>