title: How to use a for loop in Scrapy?
body: <p>I am using Scrapy for a project, in this project I am extracting the information from the xml.</p>
  
  <p>In the xml document the format where I would like to implement the for loop:</p>
  
  <pre><code>&lt;relatedPersonsList&gt;
      &lt;relatedPersonInfo&gt;...&lt;/relatedPersonInfo&gt;
      &lt;relatedPersonInfo&gt;
          &lt;relatedPersonName&gt;
              &lt;firstName&gt;Mark&lt;/firstName&gt;
              &lt;middleName&gt;E.&lt;/middleName&gt;
              &lt;lastName&gt;Lucas&lt;/lastName&gt;
          &lt;/relatedPersonName&gt;
          &lt;relatedPersonAddress&gt;
              &lt;street1&gt;1 IMATION WAY&lt;/street1&gt;
              &lt;city&gt;OAKDALE&lt;/city&gt;
              &lt;stateOrCountry&gt;MN&lt;/stateOrCountry&gt;
              &lt;stateOrCountryDescription&gt;MINNESOTA&lt;/stateOrCountryDescription&gt;
              &lt;zipCode&gt;55128&lt;/zipCode&gt;
          &lt;/relatedPersonAddress&gt;
          &lt;relatedPersonRelationshipList&gt;
              &lt;relationship&gt;Executive Officer&lt;/relationship&gt;
              &lt;relationship&gt;Director&lt;/relationship&gt;
          &lt;/relatedPersonRelationshipList&gt;
          &lt;relationshipClarification/&gt;
      &lt;/relatedPersonInfo&gt;
      &lt;relatedPersonInfo&gt;...&lt;/relatedPersonInfo&gt;
      &lt;relatedPersonInfo&gt;...&lt;/relatedPersonInfo&gt;
      &lt;relatedPersonInfo&gt;...&lt;/relatedPersonInfo&gt;
      &lt;relatedPersonInfo&gt;...&lt;/relatedPersonInfo&gt;
      &lt;relatedPersonInfo&gt;...&lt;/relatedPersonInfo&gt;
  &lt;/relatedPersonsList&gt;
  </code></pre>
  
  <p>As you can see in the <code>&lt;relatedPersonsList&gt;</code>, you can have multiple <code>&lt;relatedPersonInfo&gt;</code>, and when I try to make a for loop, I still only get the information of the first person.</p>
  
  <p>This is my actual code:</p>
  
  <pre><code>    for person in xxs.select('./relatedPersonsList/relatedPersonInfo'):
          item = Myform() #even if get rid of it I get the same result
  
          item["firstName"] = person.select('./relatedPersonName/firstName/text()').extract()[0]
          item["middleName"] = person.select('./relatedPersonName/middleName/text()')
          if item["middleName"]:
              item["middleName"] = item["middleName"].extract()[0]
          else:
              item["middleName"] = "NA"
  </code></pre>
  
  <p>here is the code that I used on my spider:</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  from scrapy.selector import XmlXPathSelector
  
  from scrapy.http import Request
  import urlparse
  from formds.items import SecformD
  
  class SecDform(CrawlSpider):
      name = "DFORM"
  
      allowed_domain = ["http://www..gov"]
      start_urls = [
          ""
      ]
  
      rules = (
  
          Rule(
              SgmlLinkExtractor(restrict_xpaths=["/html/body/div/table/tr/td[3]/a[2]"]),
              callback='parse_formd',
              #follow= True no need of follow thing
          ),
          Rule(
              SgmlLinkExtractor(restrict_xpaths=('/html/body/div/center[1]/a[contains(., "[NEXT]")]')),
              follow=True
          ),
      )
  
      def parse_formd(self, response):
          hxs = HtmlXPathSelector(response)
          sites = hxs.select('//*[@id="formDiv"]/div/table/tr[3]/td[3]/a/@href').extract()
          for site in sites:
              yield Request(url=urlparse.urljoin(response.url, site), callback=self.parse_xml_document)
  
      def parse_xml_document(self, response):
          xxs = XmlXPathSelector(response)
          item = SecformD()
          item["stateOrCountryDescription"] = xxs.select('./primaryIssuer/issuerAddress/stateOrCountryDescription/text()').extract()[0]
          item["zipCode"] = xxs.select('./primaryIssuer/issuerAddress/zipCode/text()').extract()[0]
          item["issuerPhoneNumber"] = xxs.select('./primaryIssuer/issuerPhoneNumber/text()').extract()[0]
          for person in xxs.select('./relatedPersonsList//relatedPersonInfo'):
              #item = SecDform()
  
              item["firstName"] = person.select('./relatedPersonName/firstName/text()').extract()[0]
              item["middleName"] = person.select('./relatedPersonName/middleName/text()')
              if item["middleName"]:
                  item["middleName"] = item["middleName"].extract()[0]
              else:
                  item["middleName"] = "NA"
          return item
  </code></pre>
  
  <p>I extract the information to a .json file using this command:
  scrapy crawl DFORM -o tes4.json -t json</p>