title: Writing instagram crawler with Scrapy. How can I go to the next page?
body: <p>As an exercise, I decided to write a python script that would get all the images of the specified user. I'm somewhat familiar with Scrapy, this is why I chose it as scraping tool. Currently the script is capable of downloading the images only from the first page (12 max).</p>
  
  <p>From what I can tell, instagram pages are generated by javascript. Scrapy's <code>response.body</code> (which is like source code viewed from Chrome) does not show html structure like Chrome's Inspector does. In Chrome, after 12 images, at the bottom, there's a button with link to the next page.</p>
  
  <p>For example, <a href="https://www.instagram.com/instagram/" rel="nofollow">instagram.com/instagram</a>. Link to page 2 is <a href="https://www.instagram.com/instagram/?max_id=1292385931151632610" rel="nofollow">instagram.com/instagram/?max_id=1292385931151632610</a>. On page 2 there's a link to page 3 with <code>max_id=1287301939457754444</code>.</p>
  
  <p>How can I grab that number in Scrapy so I can send my spider there? <code>response.body</code> doesn't even contain that number. Is there another way to reach the next page?</p>
  
  <p>I know Instagram API would provide some benefits but I thought it can be done without all those tokens.</p>