title: XPath: Select Certain Child Nodes
body: <p>I'm using <code>XPath</code> with <code>Scrapy</code> to scrape data off of a movie website BoxOfficeMojo.com.</p>
  
  <p>As a general question: I'm wondering how to select certain child nodes of one parent node all in one <code>Xpath</code> string.</p>
  
  <p>Depending on the movie web page from which I'm scraping data, sometimes the data I need is located at different children nodes, such as whether or not there is a link or not.  I will be going through about 14000 movies, so this process needs to be automated.</p>
  
  <p>Using <a href="http://www.boxofficemojo.com/movies/?id=actionjackson.htm" rel="nofollow">this</a> as an example.  I will need actor/s, director/s and producer/s.</p>
  
  <p>This is the <code>Xpath</code> to the director: <strong>Note</strong>: The %s corresponds to a determined index where that information is found - in the action Jackson example <code>director</code> is found at <code>[1]</code> and <code>actors</code> at <code>[2]</code>.</p>
  
  <pre><code> //div[@class="mp_box_content"]/table/tr[%s]/td[2]/font/text()
  </code></pre>
  
  <p>However, would a link exist to a page on the director, this would be the <code>Xpath</code>:</p>
  
  <pre><code> //div[@class="mp_box_content"]/table/tr[%s]/td[2]/font/a/text()
  </code></pre>
  
  <p>Actors are a bit more tricky, as there <code>&lt;br&gt;</code> included for subsequent actors listed, which may be the children of an <code>/a</code> or children of the parent <code>/font</code>, so:</p>
  
  <pre><code>//div[@class="mp_box_content"]/table/tr[%s]/td[2]/font//a/text()
  </code></pre>
  
  <p>Gets all most all of the actors (except those with <code>font/br</code>).</p>
  
  <p>Now, the main problem here, I believe, is that there are multiple <code>//div[@class="mp_box_content"]</code> - everything I have works EXCEPT that I also end up getting some digits from other <code>mp_box_content</code>.  Also I have added numerous <code>try:</code>, <code>except:</code> statements in order to get everything (actors, directors, producers who both have and do not have links associated with them).  For example, the following is my <code>Scrapy</code> code for actors:</p>
  
  <pre><code> actors = hxs.select('//div[@class="mp_box_content"]/table/tr[%s]/td[2]/font//a/text()' % (locActor,)).extract()
   try:
       second = hxs.select('//div[@class="mp_box_content"]/table/tr[%s]/td[2]/font/text()' % (locActor,)).extract()
       for n in second:
           actors.append(n)
   except:
       actors = hxs.select('//div[@class="mp_box_content"]/table/tr[%s]/td[2]/font/text()' % (locActor,)).extract()
  </code></pre>
  
  <p>This is an attempt to cover for the facts that: the first actor may not have a link associated with him/her and subsequent actors do, the first actor may have a link associated with him/her but the rest may not.</p>
  
  <p>I appreciate the time taken to read this and any attempts to help me find/address this problem!  Please let me know if any more information is needed.</p>