import requests
from scrapy.http import TextResponse

url = "http://quotes.toscrape.com/page/1/"

r = requests.get(url)
response = TextResponse(r.url, body=r.text, encoding='utf-8')

"""
Explanation: Scrapy: part 1
Scrapy is a powerful web scraping framework for Python. A framework is still a library ("an API of functions") yet with more powerful built-in features. It can be described as the combination of all we learnt till now including requests, BeautifulSoup, lxml and RegEx. To install Scrapy, open the command prompt and run the following command:
pip install scrapy
Once scrapy is installed one can start experiencing it by just running the following command inside the command prompt (e.g. let's assume you want to scrape the http://quotes.toscrape.com/page/1/ page):
scrapy shell http://quotes.toscrape.com/page/1/
Now, you must be able to apply powerful scrapy functions to get the data you want. However, all of this are available inside the command prompt. If you want to experience the same inside a Jupyter notebook, you must try to mimic the command prompt behaviour by adding 5 additional lines as shown below (instead of running the abovementioned command). As this material is provided in a Jupyter notebook, we will also mimic the command prompt behavior, yet you are encouraged to experience it yourself.
End of explanation
"""


response.css('title')

"""
Explanation: Fine, now we are ready to apply the scrapy functions on our response object. All the code following this line is same for both Jupyter notebook users and those you chose to experience the command prompt approach.
As we covered before, there are two main ways to navigate over an HTML file: using CSS selectors and the XPath approach. While BeautifulSoup supported only the former, Scrapy has functions for both: css() for using css selectors and xpath() for the xpath approach.
CSS selectors
Let's use CSS selectors to find the title of the page.
End of explanation
"""


response.css('title').extract()

"""
Explanation: As you can see it provides more information than needed. That's why there is an extract() function, that will extract only the component we are interested in without the additional information. It can be said that css() and extract() function mimic the findAll() behaviour from BeautifulSoup.
End of explanation
"""


response.css('title::text').extract()

type(response.css('title::text').extract())

"""
Explanation: Excellent! We now have the correct tag we were looking for with the text inside. If we want to choose only the text content there is no need for using additional function: one just needs to add the following component to the CSS selector ::text as shown below.
End of explanation
"""


response.css('title::text').extract_first()

type(response.css('title::text').extract_first())

"""
Explanation: As mentioned before, the extract() function applied on the css selector mimics the findAll() behavior. This is true also about the output we receive: it has the type of list. If one needs to receive the unoce element as an output, the extract_first() function must be used, which will return the very first matched element (similarly to find() from BeautifulSoup).
End of explanation
"""


response.css('h1').extract()

"""
Explanation: Let's now try to find the heading of the page (which is Quotes to Scrape). Heading is provided inside a &lt;h1&gt; tag as usually.
End of explanation
"""


response.css('h1::text').extract()

"""
Explanation: Again, we can get the heading text by using the ::text guy.
End of explanation
"""


response.css('h1 a').extract()

"""
Explanation: The latter did not really help because the heading text was inside an &lt;a&gt; tag, which in its turn was inside the above found &lt;h1&gt; tag.
End of explanation
"""


response.css('a[style="text-decoration: none"]').extract()

"""
Explanation: Nice! We found it. As you can see it has the style attribute that differenciates this &lt;a&gt; tag from others (kind of an identifier). We could use it to find this &lt;a&gt; tag even without mentioning that it is inside a &lt;h1&gt; guy. To do this in Scrapy, square brackets should be used.
End of explanation
"""


response.css('a[style="text-decoration: none"]::text').extract()

"""
Explanation: Great! Let's now extract the text first and then go for the link inside this tag (i.e. the value of the href attribute).
End of explanation
"""


response.css('a[style="text-decoration: none"]::attr(href)').extract()

"""
Explanation: To get the value of href attirubute (and same for any other attirubte) the following approach can be used in Scrapy, which can be considered the alternative to get() function in BeautifulSoup or lxml.
End of explanation
"""


# expression explanation: find Quotes, a whitespace, anything else
# return only anything else component
response.css('h1 a::text').re('Quotes\s(.*)')

"""
Explanation: Scrapy also supports regular expressions that can directly be applied on matched response. For example, let's select only the "to Scrape" part from the heading using regular expressions. We just need to substitute the extract() function with a re() function that will take the expression as an argument.
End of explanation
"""


response.css('h1 a::text').re('(\S+)\s(\S+)\s(\S+)')

"""
Explanation: Similarly, we could use RegEx to find and match and return each for of the heading separately as a list element:
End of explanation
"""


response.xpath('//title').extract()

"""
Explanation: Perfect, we are done now with css() function, let's now implement the same in xpath().
XPath approach
End of explanation
"""


response.xpath('//title/text()').extract()

"""
Explanation: To get the text only, the following should be added to the Xpath argument: /text()
End of explanation
"""


response.xpath('//h1/a').extract()

response.xpath('//h1/a/text()').extract()

"""
Explanation: Similarly, we can find the &lt;a&gt; tag inside the &lt;h1&gt; and extract first text then the link.
End of explanation
"""


response.xpath('//h1/a/@href').extract()

"""
Explanation: xpath() function operates in the same way as in the lxml package, which means /@href should be added to the path to select the value of the href attribute (i.e. the link).
End of explanation
"""


