title: Having trouble scraping a specfic site with scrapy properly
body: <p>I went over the tutorial for Scrapy, and I was able to understand how to scrap the site included in the tutorial. But I'm having a little trouble with some of the more complicated sites (at least to me).</p>
  
  <p>I'm attempting to scrape the rows and columns of the insider transactions from this webpage:</p>
  
  <p><a href="http://finviz.com/insidertrading.ashx" rel="nofollow noreferrer">http://finviz.com/insidertrading.ashx</a></p>
  
  <p>I'm using command prompt commands with scrapy to test out if I'm able to scrape the necessary information, so the following commands are what I've have written in the command prompt. </p>
  
  <pre><code>scrapy shell "http://finviz.com/insidertrading.ashx"
  </code></pre>
  
  <p>I then used firebug from firefox to look at the html code of the page.</p>
  
  <p>I'm able to get some of the information (Stock Name, Name of the Insider and Date) into a list via this code:</p>
  
  <pre><code>response.css('td a.tab-link::text').extract()
  </code></pre>
  
  <p>However, the rest of the info is missing. </p>
  
  <p>I'm able to get some (maybe most)of the missing info (Cost, Shares, Value etc) via this code</p>
  
  <pre><code>response.css(td::text).extract()
  </code></pre>
  
  <p>I can't figure out how to cleanly get all info together in one scrape. </p>
  
  <p>Thanks. </p>
  
  <p>EDIT: The other option would be to collect the data iteratively, one row at a time, so I can separate it as I like. I'm brooding over this as well. </p>