title: how can I use scrapy to extract data from webpage that has a database
body: <p>Hello I am using scrapy to accessing our intranet website and do some scraping, everything seems to be working I am able to access it, but when I am extracting data into a csv file the csv file is empty i get no errors when i run this. Each column (Directory Number, Equipment ID, Subscriber ID and etc...)from the HTML has data so how can i get that data with scrapy. </p>
  
  <pre><code>from scrapy.contrib.spiders.init import InitSpider
  from scrapy.http import Request, FormRequest
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.contrib.spiders import Rule
  
  from scrapy.spider import BaseSpider
  from scrapy.selector import HtmlXPathSelector
  
  from carrier.items import CarrierItem
  
  class CarrierSpider(InitSpider):
      name = 'dis'
      allowed_domains = ['qvpweb01.ciq.labs.att.com']
      login_page = 'https://qvpweb01.ciq.labs.att.com:8080/dis/login.jsp'
      start_urls = ["https://qvpweb01.ciq.labs.att.com:8080/dis/"]
  
      def init_request(self):
      #"""This function is called before crawling starts."""
      return Request(url=self.login_page, callback=self.login)
  
      def login(self, response):
      #"""Generate a login request."""
      return FormRequest.from_response(response,
              formdata={'txtUserName': 'xxxx', 'txtPassword': 'secret'},
              callback=self.check_login_response)
  
      def check_login_response(self, response):
      #"""Check the response returned by a login request to see if we aresuccessfully logged in."""
      if "logout" in response.body:
          self.log("\n\n\nSuccessfully logged in. Let's start crawling!\n\n\n")
          # Now the crawling can begin..
  
          return self.initialized() 
  
      else:
          self.log("\n\n\nFailed, Bad password :(\n\n\n")
          # Something went wrong, we couldn't log in, so nothing happens.
  
      def parse(self, response):
      self.log("\n\n\n We got data! \n\n\n")
      hxs = HtmlXPathSelector(response)
      sites = hxs.select('//form[@id=\'listForm\']/table/')
      items = []
      for site in sites:
          item = CarrierItem()
          item['title'] = site.select('thead/th/a/text()').extract()
          item['link'] = site.select('thead/th/a/@href').extract()
          items.append(item)
      return items
  </code></pre>
  
  <p>html from page</p>
  
  <pre><code>&lt;table width="100%" cellspacing="0" cellpadding="2" border="0"&gt;
  &lt;table width="100%" cellspacing="0" cellpadding="0" border="0"&gt;
  &lt;tbody&gt;
  &lt;tr&gt;
  &lt;td valign="top" align="left"&gt;
  &lt;html&gt;
  &lt;html&gt;
  &lt;div class="clr"&gt;&lt;/div&gt;
  &lt;table cellspacing="1" cellpadding="0" border="0"&gt;
  &lt;tbody&gt;
  &lt;tr&gt;
  &lt;tr&gt;
  &lt;td align="left" colspan="2"&gt;
  &lt;div class="list"&gt;
  &lt;form id="listForm" name="listForm" method="POST" action=""&gt;
  &lt;table&gt;
  &lt;thead&gt;
  &lt;th class="first"&gt;&lt;/th&gt;
  &lt;th&gt;
  &lt;a href="/dis/?&amp;orderby=mdn&amp;order=asc"&gt;Directory Number&lt;/a&gt;
  &lt;/th&gt;
  &lt;th&gt;
  &lt;a href="/dis/?&amp;orderby=hardwareId&amp;order=asc"&gt;Equipment ID&lt;/a&gt;
  &lt;/th&gt;
  &lt;th&gt;
  &lt;a href="/dis/?&amp;orderby=subscriberId&amp;order=asc"&gt;Subscriber ID&lt;/a&gt;
  &lt;/th&gt;
  &lt;th class="arrow_up"&gt;
  &lt;th&gt;
  &lt;a href="/dis/?&amp;orderby=sessUpldTime&amp;order=asc"&gt;Session Upload Time&lt;/a&gt;
  &lt;/th&gt;
  &lt;th&gt;
  &lt;a href="/dis/?&amp;orderby=upldRsn&amp;order=asc"&gt;Upload Reason&lt;/a&gt;
  &lt;/th&gt;
  &lt;th&gt;
  &lt;a href="/dis/?&amp;orderby=prof&amp;order=asc"&gt;Profile ID&lt;/a&gt;
  &lt;/th&gt;
  &lt;th&gt;
  &lt;img width="1" height="1" src="/dis/img/spacer.gif" alt=""&gt;
  &lt;/th&gt;
  &lt;th class="last"&gt;
  &lt;img width="1" height="1" src="/dis/img/spacer.gif" alt=""&gt;
  &lt;/th&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
  &lt;/table&gt;
  &lt;/form&gt;
  &lt;/div&gt;
  &lt;/td&gt;
  &lt;/tr&gt;
  &lt;/tbody&gt;
  &lt;/table&gt;
  &lt;/td&gt;
  &lt;/tr&gt;
  &lt;/tbody&gt;
  &lt;/table&gt;
  </code></pre>
  
  <p>output when I run the crawler</p>
  
  <pre><code>C:\Users\ye831c\Documents\Big Data\Scrapy\carrier&gt;scrapy crawl dis -o iqDis.csv
  -t csv
  2013-06-28 14:10:41-0500 [scrapy] INFO: Scrapy 0.16.5 started (bot: carrier)
  2013-06-28 14:10:41-0500 [scrapy] DEBUG: Enabled extensions: FeedExporter, LogSt
  ats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
  2013-06-28 14:10:42-0500 [scrapy] DEBUG: Enabled downloader middlewares: HttpAut
  hMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, De
  faultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMi
  ddleware, ChunkedTransferMiddleware, DownloaderStats
  2013-06-28 14:10:42-0500 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMi
  ddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddle
  ware
  2013-06-28 14:10:42-0500 [scrapy] DEBUG: Enabled item pipelines:
  2013-06-28 14:10:42-0500 [dis] INFO: Spider opened
  2013-06-28 14:10:42-0500 [dis] INFO: Crawled 0 pages (at 0 pages/min), scraped 0
   items (at 0 items/min)
  2013-06-28 14:10:42-0500 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:602
  3
  2013-06-28 14:10:42-0500 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
  2013-06-28 14:10:42-0500 [dis] DEBUG: Crawled (200) &lt;GET https://qvpweb01.ciq.la
  bs.att.com:8080/dis/login.jsp&gt; (referer: None)
  2013-06-28 14:10:42-0500 [dis] DEBUG: Redirecting (302) to &lt;GET https://qvpweb01
  .ciq.labs.att.com:8080/dis/&gt; from &lt;POST https://qvpweb01.ciq.labs.att.com:8080/d
  is/login&gt;
  2013-06-28 14:10:43-0500 [dis] DEBUG: Crawled (200) &lt;GET https://qvpweb01.ciq.la
  bs.att.com:8080/dis/&gt; (referer: https://qvpweb01.ciq.labs.att.com:8080/dis/login
  .jsp)
  2013-06-28 14:10:43-0500 [dis] DEBUG:
  
  
      Successfully logged in. Let's start crawling!
  
  
  
  2013-06-28 14:10:44-0500 [dis] DEBUG: Crawled (200) &lt;GET https://qvpweb01.ciq.la
  bs.att.com:8080/dis/&gt; (referer: https://qvpweb01.ciq.labs.att.com:8080/dis/)
  2013-06-28 14:10:44-0500 [dis] DEBUG:
  
  
       We got data!
  
  
  
  2013-06-28 14:10:44-0500 [dis] INFO: Closing spider (finished)
  2013-06-28 14:10:44-0500 [dis] INFO: Dumping Scrapy stats:
      {'downloader/request_bytes': 1382,
       'downloader/request_count': 4,
       'downloader/request_method_count/GET': 3,
       'downloader/request_method_count/POST': 1,
       'downloader/response_bytes': 146604,
       'downloader/response_count': 4,
       'downloader/response_status_count/200': 3,
       'downloader/response_status_count/302': 1,
       'finish_reason': 'finished',
       'finish_time': datetime.datetime(2013, 6, 28, 19, 10, 44, 469000),
       'log_count/DEBUG': 12,
       'log_count/INFO': 4,
       'request_depth_max': 2,
       'response_received_count': 3,
       'scheduler/dequeued': 4,
       'scheduler/dequeued/memory': 4,
       'scheduler/enqueued': 4,
       'scheduler/enqueued/memory': 4,
       'start_time': datetime.datetime(2013, 6, 28, 19, 10, 42, 15000)}
  2013-06-28 14:10:44-0500 [dis] INFO: Spider closed (finished)
  
  C:\Users\ye831c\Documents\Big Data\Scrapy\carrier&gt;
  </code></pre>