title: Interpreting callbacks and cb_kwargs with scrapy
body: <p>I'm in reach of a personal milestone with <code>scrapy</code>. The aim is to properly understand the <code>callback</code> and <code>cb_kwargs</code>, I've read the documentation countless times but I learn best with visual code, practice and an explanation.</p>
  <p>I have an example scraper, the aim is to grab the book name, price and go into each book page and extract a single piece of information. I'm trying to understand how to properly get information on the next few pages also, which I know is dependent on understanding the operation of callbacks.</p>
  <p>When I run my script It returns results only for the first page, how do I get the additional pages?</p>
  <p>Here's my scraper:</p>
  <pre><code>class BooksItem(scrapy.Item):
      items = Field(output_processor = TakeFirst())
      price = Field(output_processor = TakeFirst())
      availability = Field(output_processor = TakeFirst())
  
  class BookSpider(scrapy.Spider):
      name = &quot;books&quot;
      start_urls = ['https://books.toscrape.com']
  
      def start_request(self):
          for url in self.start_url:
              yield scrapy.Request(
                  url, 
                  callback = self.parse)
  
      def parse(self, response):
          data = response.xpath('//div[@class = &quot;col-sm-8 col-md-9&quot;]')
          for books in data:
              loader = ItemLoader(BooksItem(), selector = books)
              loader.add_xpath('items','.//article[@class=&quot;product_pod&quot;]/h3/a//text()')
              loader.add_xpath('price','.//p[@class=&quot;price_color&quot;]//text()')
              
              for url in [books.xpath('.//a//@href').get()]:
                  yield scrapy.Request(
                      response.urljoin(url),
                      callback = self.parse_book,
                      cb_kwargs = {'loader':loader})
  
          for next_page in [response.xpath('.//div/ul[@class=&quot;pager&quot;]/li[@class=&quot;next&quot;]/a//@href').get()]:
              if next_page is not None:
                  yield response.follow(next_page, callback=self.parse)
  
  
      def parse_book(self, response, loader):
          book_quote = response.xpath('//p[@class=&quot;instock availability&quot;]//text()').get()
          
  
          loader.add_value('availability', book_quote)
          yield loader.load_item()
  </code></pre>
  <p>I believe the issue is with the part where I try to grab the next few pages. I have tried an alternative approach using the following:</p>
  <pre><code>def start_request(self):
          for url in self.start_url:
              yield scrapy.Request(
                  url, 
                  callback = self.parse,
                  cb_kwargs = {'page_count':0}
  )
  
  def parse(self, response, next_page):
      if page_count &gt; 3:
          return
  ...
  ...
      page_count += 1    
      for next_page in [response.xpath('.//div/ul[@class=&quot;pager&quot;]/li[@class=&quot;next&quot;]/a//@href').get()]:
          yield response.follow(next_page, callback=self.parse, cb_kwargs = {'page_count': page_count})
  
  </code></pre>
  <p>However, I get the following error with this approach:</p>
  <blockquote>
  <p>TypeError: parse() missing 1 required positional argument: 'page_cntr'</p>
  </blockquote>