title: Scrapy POST Request with __VIEWSTATE not working javascript WebForm_PostBack
body: <p>Trying to do a simple Scrapy python script that will scrape data from a site that uses the web form likes this: </p>
  
  <p><code>onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&amp;quot;ctl00$ctl00$ctl00$ContentPlaceHolderDefault$DivMainCPH$ctl01$NRANearYouControl_2$imgbtn_Locate&amp;quot;, &amp;quot;&amp;quot;, true, &amp;quot;&amp;quot;, &amp;quot;&amp;quot;, false, false))"</code></p>
  
  <p>Let's say my site is example.com, below is my python spider that is not working but getting me back scraped HTML but not the response from the actual POST request.</p>
  
  <p>There must be a way to get back the __VIEWSTATE that asp.net uses and capture this while resending back the POST request with the proper Form Data.  I'm using Chrome debugger to try and get exactly what the request was sending.</p>
  
  <pre><code>from scrapy.spider import Spider
  from scrapy.http import FormRequest
  from scrapy.selector import Selector
  from scrapy.http import Request
  
  class ShootSpider(Spider):
      name = "shoot"
      allowed_domains = ["example.com"]
      start_urls = ["http://example.com"]
  
  def parse(self, response):
      return Request("http://findtest.example.com/",
              callback=self.parse_2)
  
  def parse_2(self, response):
     sel = Selector(response)
     self.log('A VIEW_STATE %s' % sel.xpath('//__VIEWSTATE').extract())
     self.log('A HEADER %s' % response.headers)
     self.log('A META %s' % response.meta)
     return [FormRequest(url="http://findtest.example.com/",
         formdata={'__EVENTTARGET': '', '__EVENTARGUMENT': '', '__VIEWSTATE': '',
              'ctl00%24ctl00%24ctl00%24ContentPlaceHolderDefault%24DivMainCPH%24ctl01%24NRANearYouControl_2%24LocationTextBox': 'richmond+VA',
              'ctl00%24ctl00%24ctl00%24ContentPlaceHolderDefault%24DivMainCPH%24ctl01%24NRANearYouControl_2%24ddlMiles': '200.1',
              'ctl00%24ctl00%24ctl00%24ContentPlaceHolderDefault%24DivMainCPH%24ctl01%24NRANearYouControl_2%24chkGrids%248': 'dg1',
              'ctl00%24ctl00%24ctl00%24ContentPlaceHolderDefault%24DivMainCPH%24ctl01%24NRANearYouControl_2%24imgbtn_Locate.x': '43',
              'ctl00%24ctl00%24ctl00%24ContentPlaceHolderDefault%24DivMainCPH%24ctl01%24NRANearYouControl_2%24imgbtn_Locate.y': '12'},
              callback=self.after_post)]
  
  def after_post(self, response):
      self.log('A response from %s' % response.url)
      #self.log('Body %s' % response.body)
      filename = "Example_HTML_SCRAPE"
      open(filename, 'wb').write(response.body)
  </code></pre>
  
  <p>I'm getting back HTTP Status 200 - OK after ther crawl</p>
  
  <p>Here are the command line crawl stats:</p>
  
  <p>DEBUG: Crawled (200) http://findtest.example.com/> (referer: <a href="http://findtest.example.com/" rel="nofollow">http://findtest.example.com/</a>)</p>
  
  <p>DEBUG: A response from <a href="http://findtest.example.com/" rel="nofollow">http://findtest.example.com/</a></p>
  
  <p>INFO: Closing spider (finished)</p>
  
  <p>INFO: Dumping Scrapy stats:</p>
  
  <pre><code>'downloader/request_bytes': 1451,
  'downloader/request_count': 3,
  'downloader/request_method_count/GET': 2,
   'downloader/request_method_count/POST': 1,
   'downloader/response_bytes': 100405,
   'downloader/response_count': 3,
   'downloader/response_status_count/200': 3,
   'finish_reason': 'finished',
   'finish_time': datetime.datetime(2014, 4, 24, 2, 42, 42, 766547),
   'log_count/DEBUG': 9,
   'log_count/INFO': 7,
   'request_depth_max': 2,
   'response_received_count': 3,
   'scheduler/dequeued': 3,
   'scheduler/dequeued/memory': 3,
   'scheduler/enqueued': 3,
   'scheduler/enqueued/memory': 3,
   'start_time': datetime.datetime(2014, 4, 24, 2, 42, 41, 381395)}
  </code></pre>