title: BeautifulSoup: How do I extract all the <li>s from a list of <ul>s that contains some nested <ul>s?
body: <p>I'm a newbie programmer trying to jump in to Python by building a script that scrapes <a href="http://en.wikipedia.org/wiki/2000s_in_film" rel="nofollow noreferrer">http://en.wikipedia.org/wiki/2000s_in_film</a> and extracts a list of &quot;Movie Title (Year)&quot;.
  My HTML source looks like:</p>
  <pre><code>&lt;h3&gt;Header3 (Start here)&lt;/h3&gt;
  &lt;ul&gt;
      &lt;li&gt;List items&lt;/li&gt;
      &lt;li&gt;Etc...&lt;/li&gt;
  &lt;/ul&gt;
  &lt;h3&gt;Header 3&lt;/h3&gt;
  &lt;ul&gt;
      &lt;li&gt;List items&lt;/li&gt;
      &lt;ul&gt;
          &lt;li&gt;Nested list items&lt;/li&gt;
          &lt;li&gt;Nested list items&lt;/li&gt;&lt;/ul&gt;
      &lt;li&gt;List items&lt;/li&gt;
  &lt;/ul&gt;
  &lt;h2&gt;Header 2 (end here)&lt;/h2&gt;
  </code></pre>
  <p>I'd like all the <code>li</code> tags following the first <code>h3</code> tag and stopping at the next <code>h2</code> tag, including all nested <code>li</code> tags.</p>
  <pre><code>firstH3 = soup.find('h3')
  </code></pre>
  <p>...correctly finds the place I'd like to start.</p>
  <pre><code>firstH3 = soup.find('h3') # Start here
  uls = []
  for nextSibling in firstH3.findNextSiblings():
      if nextSibling.name == 'h2':
          break
      if nextSibling.name == 'ul':
          uls.append(nextSibling)
  </code></pre>
  <p>...gives me a list <code>uls</code>, each with <code>li</code> contents that I need.</p>
  <p>Excerpt of the <code>uls</code> list:</p>
  <pre><code>&lt;ul&gt;
  ...
      &lt;li&gt;&lt;i&gt;&lt;a href=&quot;/wiki/Agent_Cody_Banks&quot; title=&quot;Agent Cody Banks&quot;&gt;Agent Cody Banks&lt;/a&gt;&lt;/i&gt; (2003)&lt;/li&gt;
      &lt;li&gt;&lt;i&gt;&lt;a href=&quot;/wiki/Agent_Cody_Banks_2:_Destination_London&quot; title=&quot;Agent Cody Banks 2: Destination London&quot;&gt;Agent Cody Banks 2: Destination London&lt;/a&gt;&lt;/i&gt; (2004)&lt;/li&gt;
      &lt;li&gt;Air Bud series:
          &lt;ul&gt;
              &lt;li&gt;&lt;i&gt;&lt;a href=&quot;/wiki/Air_Bud:_World_Pup&quot; title=&quot;Air Bud: World Pup&quot;&gt;Air Bud: World Pup&lt;/a&gt;&lt;/i&gt; (2000)&lt;/li&gt;
              &lt;li&gt;&lt;i&gt;&lt;a href=&quot;/wiki/Air_Bud:_Seventh_Inning_Fetch&quot; title=&quot;Air Bud: Seventh Inning Fetch&quot;&gt;Air Bud: Seventh Inning Fetch&lt;/a&gt;&lt;/i&gt; (2002)&lt;/li&gt;
              &lt;li&gt;&lt;i&gt;&lt;a href=&quot;/wiki/Air_Bud:_Spikes_Back&quot; title=&quot;Air Bud: Spikes Back&quot;&gt;Air Bud: Spikes Back&lt;/a&gt;&lt;/i&gt; (2003)&lt;/li&gt;
              &lt;li&gt;&lt;i&gt;&lt;a href=&quot;/wiki/Air_Buddies&quot; title=&quot;Air Buddies&quot;&gt;Air Buddies&lt;/a&gt;&lt;/i&gt; (2006)&lt;/li&gt;
          &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;i&gt;&lt;a href=&quot;/wiki/Akeelah_and_the_Bee&quot; title=&quot;Akeelah and the Bee&quot;&gt;Akeelah and the Bee&lt;/a&gt;&lt;/i&gt; (2006)&lt;/li&gt;
  ...
  &lt;/ul&gt;
  </code></pre>
  <p>But I'm unsure of where to go from here.</p>
  <hr />
  <p>Update:</p>
  <p><strong>Final Code:</strong></p>
  <pre><code>lis = []
      for ul in uls:
          for li in ul.findAll('li'):
              if li.find('ul'):
                  break
              lis.append(li)
  
      for li in lis:
          print li.text.encode(&quot;utf-8&quot;)
  </code></pre>
  <p>The <code>if</code>...<code>break</code> throws out the LI's that contain UL's since the nested LI's are now duplicated.</p>
  <p>Print output is now:</p>
  <blockquote>
  <ul>
  <li>102 Dalmatians(2000)</li>
  <li>10th &amp; Wolf(2006)</li>
  <li>11:14(2006)</li>
  <li>12:08 East of Bucharest(2006)</li>
  <li>13 Going on 30(2004)</li>
  <li>1408(2007)</li>
  <li>...</li>
  </ul>
  </blockquote>