title: Beautiful Soup iterate over html tag
body: <p>I have the following code in html</p>
  
  <p><div class="snippet" data-lang="js" data-hide="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>&lt;section&gt;
      &lt;section&gt;
          &lt;h2&gt;Title1&lt;/h2&gt;
          &lt;p&gt;Text1&lt;/p&gt;
          &lt;p&gt;Text1&lt;/p&gt;
       &lt;/section&gt;
    &lt;section&gt;
          &lt;h2&gt;Title2&lt;/h2&gt;
          &lt;p&gt;Text2&lt;/p&gt;
          &lt;p&gt;Text2&lt;/p&gt;
       &lt;/section&gt;
    &lt;section&gt;
          &lt;h2&gt;Title3&lt;/h2&gt;
          &lt;p&gt;Text3&lt;/p&gt;
          &lt;p&gt;Text3&lt;/p&gt;
       &lt;/section&gt;
    &lt;/section&gt;
  &lt;section&gt;
          &lt;h2&gt;Title2-1&lt;/h2&gt;
          &lt;p&gt;Text2-1&lt;/p&gt;
          &lt;p&gt;Text2-1&lt;/p&gt;
  &lt;/section&gt;
  &lt;section&gt;
          &lt;h2&gt;Title3-1&lt;/h2&gt;
          &lt;p&gt;Text3-1&lt;/p&gt;
          &lt;p&gt;Text3-1&lt;/p&gt;
  &lt;/section&gt;</code></pre>
  </div>
  </div>
  
  As in some sections have subsections and some don't. 
  I want to grab the content of sub-sections and sections that don't have subsections
  I am trying to iterate over the subsections, so that I can create an Index in scrapy.
  I have following code for scrapy:</p>
  
  <pre><code>class RUSpider(BaseSpider):
  name = "ru"
  allowed_domains = ["http://127.0.0.1:8000/"]
  start_urls = [
      "http://127.0.0.1:8000/week2/1_am/#/",
      "http://127.0.0.1:8000/week1/1/",
      "http://127.0.0.1:8000/week3/1_am/"
  ]
  rules = [
      Rule(SgmlLinkExtractor(), follow=True)
  ]
  
  def parse(self, response):
  
      filename = response.url.split("/")[3]
      hxs = HtmlXPathSelector(response)
      divs = hxs.select('//div')
      sections = divs.select('//section').extract()
     # print sections.extract
  
  
   #class definition for scrapy and html selector
  
      for each in sections: #iterate over loop [above sections]
          soup = BeautifulSoup(each)
          sp= soup.prettify()
          elements = soup.findAll("section".split())
          print len(elements),'sublength'
          if len(elements ) &gt; 1:
              for element in elements:
                  for subelement in element:
                      print subelement,'element'
          else:
              item = RItem() # create Index Item
              item['html_content'] = each
              print each
              yield item
  </code></pre>
  
  <p>Some of the result is correctly formatted, although some sections without subsections are getting broken down into individual elements.</p>
  
  <p>I want each section individually. I mean since 1 section has other sections. I want to loop over these sections and get them individually so I can keep track of loop. Since some sections don't have sub-sections no need to loop through them.</p>
  
  <p>Is there a better way of doing this in BeautifulSoup?
  I want the following output</p>
  
  <p><div class="snippet" data-lang="js" data-hide="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>    &lt;section&gt;
          &lt;h2&gt;Title1&lt;/h2&gt;
          &lt;p&gt;Text1&lt;/p&gt;
          &lt;p&gt;Text1&lt;/p&gt;
       &lt;/section&gt;
    &lt;section&gt;
          &lt;h2&gt;Title2&lt;/h2&gt;
          &lt;p&gt;Text2&lt;/p&gt;
          &lt;p&gt;Text2&lt;/p&gt;
       &lt;/section&gt;
    &lt;section&gt;
          &lt;h2&gt;Title3&lt;/h2&gt;
          &lt;p&gt;Text3&lt;/p&gt;
          &lt;p&gt;Text3&lt;/p&gt;
       &lt;/section&gt;
   
      &lt;section&gt;
              &lt;h2&gt;Title2-1&lt;/h2&gt;
              &lt;p&gt;Text2-1&lt;/p&gt;
              &lt;p&gt;Text2-1&lt;/p&gt;
      &lt;/section&gt;
      &lt;section&gt;
              &lt;h2&gt;Title3-1&lt;/h2&gt;
              &lt;p&gt;Text3-1&lt;/p&gt;
              &lt;p&gt;Text3-1&lt;/p&gt;
      &lt;/section&gt;</code></pre>
  </div>
  </div>
  </p>