title: Not able to scrape the description properly using Beautiful Soup and Python
body: <p>I am web-scraping this link : <a href="https://www.americanexpress.com/in/credit-cards/smart-earn-credit-card/?linknav=in-amex-cardshop-allcards-learn-SmartEarnCreditCard-carousel" rel="nofollow noreferrer">https://www.americanexpress.com/in/credit-cards/smart-earn-credit-card/?linknav=in-amex-cardshop-allcards-learn-SmartEarnCreditCard-carousel</a>
  using bs4 and python.</p>
  <p>I am basically grabbing the key benefits from that website using the following code.</p>
  <pre><code>link = 'https://www.americanexpress.com/in/credit-cards/smart-earn-credit-card/?linknav=in-amex-cardshop-allcards-learn-SmartEarnCreditCard-carousel'
  html = urlopen(link)
  soup = BeautifulSoup(html, 'lxml')
  
  details = []
  
  for span in soup.select(&quot;.why-amex__subtitle span&quot;):
      details.append(f'{span.get_text(strip=True)}: {span.find_next(&quot;span&quot;).get_text(strip=True)}')
  
  
  
  print(details)
  
  </code></pre>
  <p><strong>Output</strong></p>
  <pre><code>['Accelerated Earn Rate: Earn 10X Membership Rewards® Points2on your spending on Flipkart and Uber and earn 5X Membership Rewards Points2on Amazon, Swiggy, BookMyShow and more.', 'Welcome Bonus: Rs. 500 cashback as Welcome Gift on eligible spends1of Rs. 10,000 in the first 90 days of Cardmembership', 'Renewal Fee Waiver: Get a renewal fee waiver on eligible spends3of Rs.40,000 and above in the previous year of Cardmembership', 'AMERICAN EXPRESS EMI: Convert purchases into']
  
  
  </code></pre>
  <p>The last item in this list is not scraped properly, it is incomplete.
  Because there is a hyperlink in the middle of the text.</p>
  <p>Below is the html code corresponding to that problem:</p>
  <pre><code>&lt;div class=&quot;why-amex__col&quot;&gt;&lt;span class=&quot;icons  why-amex__lrgIcon icon-Amex-Icons-2016-85&quot;&gt;&lt;/span&gt;&lt;h4 class=&quot;why-amex__subtitle&quot;&gt;&lt;div&gt;&lt;span&gt;AMERICAN EXPRESS EMI&lt;/span&gt;&lt;/div&gt;&lt;/h4&gt;&lt;div class=&quot;why-amex__copy&quot;&gt;&lt;div class=&quot;description_text&quot;&gt;&lt;div&gt;&lt;span&gt;Convert purchases into &lt;/span&gt;&lt;a href=&quot;https://www.americanexpress.com/india/membershiprewards/cardmember_offers/viewmore.html&quot; target=&quot;_blank&quot;&gt;EMI&lt;/a&gt;&lt;span&gt; at the point of sale with an interest rate as low as 12% p.a. and zero foreclosure charges&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;
  </code></pre>
  <p>I'd like to get the full description of the last item without missing out the text.</p>