Scraping a webpage copying with the logic of scrapy Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be applied in regular method in python. 

So, I tried to make one. Here is what I did. My scraper opens a webpage, parses the 10 tags from it's right sided area then tracks down each tags and then again follows its pagination and parses the whole content. Hope I did it the right way. Here is the code:

import requests ; from lxml import html

core_link = "http://quotes.toscrape.com/"

def quotes_scraper(base_link):
    response = requests.get(base_link)
    tree = html.fromstring(response.text)
    for titles in tree.cssselect("span.tag-item a.tag"):
        processing_docs(core_link + titles.attrib['href'])

def processing_docs(base_link):
    response = requests.get(base_link).text
    root = html.fromstring(response)
    for soups in root.cssselect("div.quote"):
        quote = soups.cssselect("span.text")[0].text
        author = soups.cssselect("small.author")[0].text
        print(quote, author)

    next_page = root.cssselect("li.next a")[0].attrib['href'] if root.cssselect("li.next a") else ""
    if next_page:
        page_link = core_link + next_page
        processing_docs(page_link)

quotes_scraper(core_link)
Even though you are trying to mimic what Scrapy spider might look like, there is a very major high-level difference between how your code is executed and how a Scrapy spider is.

Scrapy is entirely asynchronous since it is based on a twisted network library which makes the code operate in a non-blocking nature, to quote the documentation:


  Requests are scheduled and processed asynchronously. This means that
  Scrapy doesn’t need to wait for a request to be finished and
  processed, it can send another request or do other things in the
  meantime. This also means that other requests can keep going even if
  some request fails or an error happens while handling it.


But, in your case, you do go through the whole process in a blocking mode, processing requests one by one:


Extract tag links from the main page
Make a request to the next tag in the list and wait for the response
Extract quotes and print them
Extract the link to the next page and go to step number 2^


Every time requests make a request, you wait. Scrapy would do things while waiting, that's the main high-level difference.

And, by the way, if you would not assign priority values for your Scrapy requests (by default all requests have the same priority), there is no enforced order in which requests are gonna be processed.