title: Issue with scraping data using beautiful soup
body: <p>I am using following code to scrape data from a website.</p>
  
  <pre><code># -*- coding: cp1252 -*-
  import urllib2
  import sys
  from bs4 import BeautifulSoup
  
  page = urllib2.urlopen('http://www.att.com/shop/wireless/plans-new.html#fbid=U-XD_DHOGEp').read()
  soup = BeautifulSoup(page)
  plans = soup.findAll('div', {"class": "planTitle"})
  for plan in plans:
      planname = u' '.join(plan.stripped_strings)
      plantypes = soup.findAll('div', {"class":"top"})
      prices = soup.findAll('div', {"class":"bottom"})
      for plantype, price in zip(plantypes, prices):
          plantype1 = u' '.join(plantype.stripped_strings)
          price1 = u' '.join(price.stripped_strings)
          print planname, plantype1, price1
  </code></pre>
  
  <p>Problem: If you go through the webpage which is mentioned in this code, these are 4-5 types of plans and for each plan 3 voice options and some 2-3 data options are present. I want to scrape the data in such way that for each plan I can get its respective voice options against it and then monthly prices for those options.</p>
  
  <p>Code which I am running right now returns the all possible combinations of plan name + voice options. For each plan name I get some 20-30 entries because it creates a entry even for wrong Plan name + voice options combination. Eg. Individual Plan - 550 minutes - $59.99, in this combination 500 minutes and 59.99 are part of Family Plan.</p>
  
  <p>I want the loop to run so that only correct Plan + Voice options combination is extracted.</p>
  
  <p>Snippet of Webpage:
  For each plan one box is present on webpage, which contains voice options and prices corresponding to those options, I want the loop to run for each box, but element + class combination for voice options and their prices is not unique. That's why plan name takes value from other boxex also.</p>
  
  <pre><code>&lt;div class="innerContainer"&gt; 
  
      &lt;div class="planTitle"&gt; 
          &lt;h2&gt;&lt;a href="http://www.att.com/shop/wireless/plans/individualplans.html" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;AT&amp;amp;T Individual Plans&lt;/a&gt;&lt;/h2&gt; 
      &lt;/div&gt; 
      &lt;div class="planSubTitle"&gt; 
          &lt;img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-clock.jpg" alt=""&gt; 
          &lt;p&gt;Voice plan options:&lt;/p&gt; 
      &lt;/div&gt; 
      &lt;!-- Begin three white boxes --&gt; 
      &lt;!-- Note, extra boxes can be added to the row with the following method  --&gt; 
      &lt;!-- 1. Add more div containers inside .whiteBox  --&gt; 
      &lt;!-- 2. Modify class names to boxes_one, boxes_two, boxes_three etc... (max six) --&gt; 
      &lt;div class="whiteBox"&gt; 
          &lt;div class="boxes_three"&gt; 
              &lt;a class="lnk-help tooltips fullBoxLink" href="#smartphone_individual_voice_450" onclick="window.location.href = 'http://www.att.com/shop/wireless/plans/voice/sku3830290.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spindvoice450';return false;" aria-describedby="smartphone_individual_voice_450" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010" title=""&gt;&lt;/a&gt; 
              &lt;span id="smartphone_individual_voice_450" class="tips" role="tooltip"&gt;$0.45/min. for additional minutes&lt;/span&gt; 
              &lt;div class="top"&gt; 
                  &lt;p class="stat"&gt;450&lt;/p&gt; 
                  &lt;p class="statText"&gt;Minutes&lt;/p&gt; 
              &lt;/div&gt; 
              &lt;div class="bottom"&gt; 
                  &lt;p&gt;$39.99/mo.&lt;/p&gt; 
              &lt;/div&gt; 
          &lt;/div&gt; 
          &lt;div class="boxes_three"&gt; 
              &lt;a class="lnk-help tooltips fullBoxLink" href="#smartphone_individual_voice_900" onclick="window.location.href = 'http://www.att.com/shop/wireless/plans/voice/sku3830292.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spindvoice900';return false;" aria-describedby="smartphone_individual_voice_900" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010" title=""&gt;&lt;/a&gt; 
              &lt;span id="smartphone_individual_voice_900" class="tips" role="tooltip"&gt;$0.40/min. for additional minutes&lt;/span&gt; 
              &lt;div class="top"&gt; 
                  &lt;p class="stat"&gt;900&lt;/p&gt; 
                  &lt;p class="statText"&gt;Minutes&lt;/p&gt; 
              &lt;/div&gt; 
              &lt;div class="bottom"&gt; 
                  &lt;p&gt;$59.99/mo.&lt;/p&gt; 
              &lt;/div&gt; 
          &lt;/div&gt; 
          &lt;div class="boxes_three borderNone"&gt; 
              &lt;a class="fullBoxLink" href="http://www.att.com/shop/wireless/plans/voice/sku3830293.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spindvoiceunlim" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;&lt;/a&gt; 
              &lt;div class="top"&gt; 
                  &lt;p class="stat"&gt;Unlimited&lt;/p&gt; 
                  &lt;p class="statText"&gt;Minutes&lt;/p&gt; 
              &lt;/div&gt; 
              &lt;div class="bottom"&gt; 
                  &lt;p&gt;$69.99/mo.&lt;/p&gt; 
              &lt;/div&gt; 
          &lt;/div&gt; 
      &lt;/div&gt; 
      &lt;!-- End three white boxes --&gt; 
      &lt;!-- Begin left gray container --&gt; 
      &lt;div class="containerTwoThirds"&gt; 
          &lt;div class="planSubTitle"&gt; 
              &lt;img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-globe.jpg" alt=""&gt; 
              &lt;p&gt;Data plan options:&lt;/p&gt; 
          &lt;/div&gt; 
          &lt;div class="grayTwoThirds"&gt; 
              &lt;div class="grayBox"&gt; 
                  &lt;a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/dataplus300mb-smartphone4glte-sku5380269.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spinddata300mb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;&lt;/a&gt; 
                  &lt;p class="stat"&gt;&lt;strong&gt;300MB&lt;/strong&gt;&lt;/p&gt; 
                  &lt;p class="statText"&gt;$20.00/mo.&lt;/p&gt; 
              &lt;/div&gt; 
              &lt;div class="grayBoxBreak"&gt;&lt;/div&gt; 
              &lt;div class="grayBox"&gt; 
                  &lt;a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/datapro3gb-smartphone4glte-sku5470232.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spinddata3gb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;&lt;/a&gt; 
                  &lt;p class="stat"&gt;&lt;strong&gt;3GB&lt;/strong&gt;&lt;/p&gt; 
                  &lt;p class="statText"&gt;$30.00/mo.&lt;/p&gt; 
              &lt;/div&gt; 
              &lt;div class="grayBoxBreak"&gt;&lt;/div&gt; 
              &lt;div class="grayBox"&gt; 
                  &lt;a class="fullBoxLink" href="http://www.att.com/shop/wireless/services/datapro5gb-smartphone4glte-sku5480228.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spinddata5gb" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;&lt;/a&gt; 
                  &lt;p class="stat"&gt;&lt;strong&gt;5GB&lt;/strong&gt;&lt;/p&gt; 
                  &lt;p class="statText"&gt;$50.00/mo.&lt;/p&gt; 
              &lt;/div&gt; 
          &lt;/div&gt; 
      &lt;/div&gt; 
      &lt;!-- End left gray container --&gt; 
      &lt;!-- Begin right gray container --&gt; 
      &lt;div class="containerThird"&gt; 
          &lt;div class="planSubTitle"&gt; 
              &lt;img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/tiny-phone.jpg" alt=""&gt; 
              &lt;p&gt;Messaging plan options: &lt;span class="fix"&gt;&lt;/span&gt;&lt;/p&gt; 
          &lt;/div&gt; 
          &lt;div class="grayThird"&gt; 
              &lt;div class="grayBox"&gt;  
                  &lt;a data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2012325" href="http://www.att.com/shop/wireless/services/messagingunlimited-sku1160055.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spindmessunlim" class="fullBoxLink"&gt;&lt;/a&gt;  
                  &lt;p class="stat"&gt;&lt;strong&gt;ULTD&lt;/strong&gt; MSGS&lt;/p&gt;  
                  &lt;p class="statText"&gt;$20.00/mo.&lt;/p&gt;  
              &lt;/div&gt; 
              &lt;div class="grayBoxBreak"&gt;&lt;/div&gt; 
              &lt;div class="grayBox last"&gt; 
                  &lt;p class="stat"&gt;&lt;strong&gt;PAY PER USE&lt;/strong&gt;&lt;/p&gt; 
                  &lt;p class="statText"&gt;20¢/text &lt;span class="lightGray"&gt;|&lt;/span&gt; 30¢/pic/video&lt;/p&gt; 
              &lt;/div&gt; 
          &lt;/div&gt; 
      &lt;/div&gt; 
      &lt;!-- End right gray container --&gt; 
      &lt;!-- Begin sub footer --&gt; 
      &lt;div class="bottomLinks"&gt;  
          &lt;div class="links"&gt; 
              &lt;a href="http://www.att.com/shop/wireless/plans/individualplans.html?taxoPlan=POSTPAID-INDIVIDUAL-CANADA&amp;amp;source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spindcanada" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;Nation with Canada Plans&lt;/a&gt; | &lt;a href="http://www.att.com/shop/wireless/plans/voice/sku5740279.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spindhomephone" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;Unlimited Home Phone&lt;/a&gt; | &lt;a href="http://www.att.com/shop/wireless/plans/voice/sku3830294.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=spindsenior" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;Senior Plans&lt;/a&gt; 
          &lt;/div&gt; 
          &lt;a class="shop_button" href="http://www.att.com/shop/wireless/devices/smartphones.html?source=IC95ATPLP00PSP00L&amp;amp;wtExtndSource=indshopsp" data-cqpath="/content/att/shop/en/wireless/plans-new/jcr:content/maincontent/authortext;2013010"&gt;&lt;img src="/shopcms/media/att/2012/shop/wireless/promotions/Plans-CM_page/buttons/shop_smartphones.png" alt="Shop Smartphones" width="158" height="29"&gt;&lt;/a&gt; 
      &lt;/div&gt; 
      &lt;!-- End sub footer --&gt; 
  &lt;/div&gt;
  </code></pre>
  
  <p>Please help me in solving this issue as I am new to programming.</p>