title: scrapy formrequest producing mismatched and missing results
body: <p>I am running a series of formrequests in scrapy, jumping from one page to the next, while going from one row to the next within a page, and scraping document numbers and names from each row.</p>
  <p>However, the scraped data when outputted to a csv seems to appear mismatched and sometimes missing. Any idea why this is the case?</p>
  <pre><code>    formdata_pre={
              'ScriptManager1' : &quot;SearchFormEx1$UpdatePanel|SearchFormEx1$btnSearch&quot;,
              'ScriptManager1_HiddenField': '', 
              'Navigator1$SearchOptions1$SavePrintCriteriaCheck' : 'on',
              'Navigator1$SearchOptions1$SaveOrderCriteriaCheck' : 'on',
              'SearchCriteriaOffice1$DDL_OfficeName' : 'Recorded Land',
              'SearchCriteriaName1$DDL_SearchName' : 'Recorded Land Name Search',
              'SearchFormEx1$ACSTextBox_LastName1' : 'mortgage electronic',
              'SearchFormEx1$ACSTextBox_FirstName1' : '',
              'SearchFormEx1$ACSRadioButtonList_PartyType1' : '',
              'SearchFormEx1$ACSTextBox_LastName2' : '',
              'SearchFormEx1$ACSTextBox_FirstName2' : '',
              'SearchFormEx1$ACSRadioButtonList_PartyType2' : '',
              'SearchFormEx1$ACSRadioButtonList_Search' : '3',
              'SearchFormEx1$ACSDropDownList_DocumentType' : '29',
              'SearchFormEx1$ACSDropDownList_Towns' : '-2',
              'SearchFormEx1$ACSTextBox_DateFrom' : '1/1/1753',
              'SearchFormEx1$ACSTextBox_DateTo' : '10/14/2000',
              'ImageViewer1$ScrollPos' : '',
              'ImageViewer1$ScrollPosChange' : '',
              'ImageViewer1$_imgContainerWidth' : '0',
              'ImageViewer1$_imgContainerHeight' : '0',
              'ImageViewer1$isImageViewerVisible' : 'true',
              'ImageViewer1$hdnWidgetSize' : '',
              'ImageViewer1$DragResizeExtender_ClientState' : '',
              'CertificateViewer1$ScrollPos' : '',
              'CertificateViewer1$ScrollPosChange' : '',
              'CertificateViewer1$_imgContainerWidth' : '0',
              'CertificateViewer1$_imgContainerHeight' : '0',
              'CertificateViewer1$isImageViewerVisible' : 'true',
              'CertificateViewer1$hdnWidgetSize' : '',
              'CertificateViewer1$DragResizeExtender_ClientState' : '',
              'PTAXViewer1$ScrollPos' : '',
              'PTAXViewer1$ScrollPosChange' : '',
              'PTAXViewer1$_imgContainerWidth' : '0',
              'PTAXViewer1$_imgContainerHeight' : '0',
              'PTAXViewer1$isImageViewerVisible' : 'true',
              'PTAXViewer1$hdnWidgetSize' : '',
              'PTAXViewer1$DragResizeExtender_ClientState' : '',
              'DocList1$ctl12' : '',
              'DocList1$ctl14' : '',
              'RefinementCtrl1$ctl01' : '',
              'RefinementCtrl1$ctl03' : '',
              'NameList1$ScrollPos' : '',
              'NameList1$ScrollPosChange' : '',
              'NameList1$_SortExpression' : '',
              'NameList1$ctl03' : '',
              'NameList1$ctl05' : '',
              'DocDetails1$PageSize' : '',
              'DocDetails1$PageIndex' : '',
              'DocDetails1$SortExpression' : '',
              'BasketCtrl1$ctl01' : '',
              'BasketCtrl1$ctl03' : '',
              'OrderList1$ctl01' : '',
              'OrderList1$ctl03' : '',
              '__EVENTTARGET' : '',
              '__EVENTARGUMENT' : '',
              '__LASTFOCUS' : '',
              '__VIEWSTATE' : '',
              '__ASYNCPOST' : 'true',
              'SearchFormEx1$btnSearch' : 'Search'
              }
  </code></pre>
  <p>Above is the formdata_pre for the formdata below--stack overflow is being weird about this dict.</p>
  <pre><code>import scrapy
  from scrapy import FormRequest
  from scrapy.shell import inspect_response
  import re
  
  class FormySpider(scrapy.Spider):
  
      name = 'formy'
      allowed_domains = ['i2a.uslandrecords.com']
      start_urls = ['https://i2a.uslandrecords.com/ME/Cumberland/D/Default.aspx']
      def parse(self, response):
          URL = 'https://i2a.uslandrecords.com/ME/Cumberland/D/Default.aspx'
          headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'}        
          yield FormRequest(url=URL, 
              method='POST',
              headers = headers,
              formdata={
              'ScriptManager1' : 'Navigator1$SearchOptions1$UpdatePanel|Navigator1$SearchOptions1$DocImagesCheck',
              '__EVENTTARGET' : 'Navigator1$SearchOptions1$DocImagesCheck',
              '__ASYNCPOST' : ' true'},
              dont_filter=True,
              callback=self.after_login_1
              )
  
      def after_login_1(self, response):        
          URL = 'https://i2a.uslandrecords.com/ME/Cumberland/D/Default.aspx'
          headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'}        
          formdata=formdata_pre,
          yield FormRequest(url=URL,
              method='POST',
              headers = headers,
              formdata=formdata,
              dont_filter=True,
              callback=self.after_login_2
              )
  
      def after_login_2(self, response):
          URL = 'https://i2a.uslandrecords.com/ME/Cumberland/D/Default.aspx'
          headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'}        
          if response.xpath(&quot;(//a[contains(@id,'ButtonRow_Doc')])[1]&quot;).get():            print(response.xpath(&quot;//a[contains(@id,'LinkButton')]/text()&quot;).getall())            print(response.xpath(&quot;//a[contains(@id,'ButtonRow_Doc')]/text()&quot;).getall())
              for row in response.xpath(&quot;//a[contains(@id,'ButtonRow_Doc')]/@href&quot;).getall():
                  event =re.findall(r'doPostBack\(\'([^()].*)\'\,\'\'\)',row)[0]
                  scriptmanager = 'DocList1$UpdatePanel|'+event
                  ndoc=re.findall(r'\#\_(.*)\'\,',row)[0]
                  nbook = response.xpath( &quot;(//a[contains(@href,\&quot;_{}',\&quot;)])[3]/text()&quot;.format(ndoc)).get()                
                  npage = response.xpath( &quot;(//a[contains(@href,\&quot;_{}',\&quot;)])[4]/text()&quot;.format(ndoc)).get()
                  yield FormRequest(url=URL, 
                      method='POST',
                      headers = headers,
                      formdata={
                      'ScriptManager1' : scriptmanager,
                      '__EVENTTARGET' : event,
                      '__ASYNCPOST' : ' true'},
                      dont_filter=True,
                      meta = {
                      'nbook' : nbook,
                      'npage' : npage
                      },
                      callback=self.after_login_3
                      )                
                  if response.xpath(&quot;//a[@id='DocList1_LinkButtonNext']&quot;).get():            
  
                      yield FormRequest(url=URL, 
                          method='POST',
                          headers = headers,
                          formdata={
                          'ScriptManager1' : 'DocList1$UpdatePanel|DocList1$LinkButtonNext',
                          '__EVENTTARGET' : 'DocList1$LinkButtonNext',
                          '__ASYNCPOST' : ' true'},
                          dont_filter=True,
                          callback=self.after_login_2
                          )
                  else:            
                      print(&quot;empty week&quot;)
  
      def after_login_3(self, response):
          URL = 'https://i2a.uslandrecords.com/ME/Cumberland/D/Default.aspx'
          headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'}
          names  = response.xpath(&quot;//a[contains(@id,'GrantorGrantee')]/text()&quot;).getall()
          nbook = response.request.meta['nbook']
          npage = response.request.meta['npage']
          yield {        
          'nbook' : nbook,
          'npage' : npage,
          'names' : names
          }
  </code></pre>
  <p>In terms of output, here is an example of a mismatch:</p>
  <p>15643 ||    20  || {FRANTZ RICHARD C  ,MORTGAGE ELECTRONIC REGISTRATION SYSTEMS INC   ,FLEET MORTGAGE CORP}</p>
  <p>In the above, the names do not match with the page and book numbers and the name &quot;HUGHES ANDREW&quot; is entirely absent. This is just one example.</p>
  <p>I think that the scrapy request is going too fast for the response to produce, and maybe this lag is creating the disconnect and missing objects.</p>