Parser for a lazyloading yellowpage I've written a script to harvest Name and Phone number from yellowpage Canada under pizza category. The web page doesn't show its full content until scrolled downmost. In my crawler's every cycle, it fetches 40 records. I twitched a little in my loop which I've learnt lately to get the full content. Hope I did it the way it should be.

import requests
from lxml import html

Base_url="https://www.yellowpages.ca/search/si/{0}/pizza/Toronto"
with requests.session() as session:
    page_num=1
    while True:
        response = session.get(Base_url.format(page_num))
        if response.status_code==404:
            break
        tree = html.fromstring(response.text)
        for titles in tree.xpath("//div[@itemprop='itemListElement']"):
            try:
                title = titles.xpath(".//h3[@itemprop='name']/a/text()")[0]
            except IndexError:
                title=""
            try:
                phone= titles.xpath(".//h4[@itemprop='telephone']/text()")[0]
            except IndexError:
                phone=""
            print(title,phone)
        page_num+=1
Applying PEP8 recommendations about naming and formatting,
as well as wrapping the main logic in a main function and calling from within a if __name__ == '__main__': guard,
the code would become easier to read and possible to import and test:

import requests
from lxml import html

url_format = "https://www.yellowpages.ca/search/si/{0}/pizza/Toronto"


def main():
    with requests.session() as session:
        page_num = 1
        while True:
            response = session.get(url_format.format(page_num))
            if response.status_code == 404:
                break
            tree = html.fromstring(response.text)
            for titles in tree.xpath("//div[@itemprop='itemListElement']"):
                try:
                    title = titles.xpath(".//h3[@itemprop='name']/a/text()")[0]
                except IndexError:
                    title = ""
                try:
                    phone = titles.xpath(".//h4[@itemprop='telephone']/text()")[0]
                except IndexError:
                    phone = ""
                print(title, phone)
            page_num += 1


if __name__ == '__main__':
    main()


I'm not a huge fan of using exceptions in situations that are not really exceptions, like here:


try:
    title = titles.xpath(".//h3[@itemprop='name']/a/text()")[0]
except IndexError:
    title = ""



What if a titles.xpath(...) call would raise an IndexError due to a bug? It would be inadvertently caught and unnoticed.
Catching IndexError here is an indirect way of the real intention,
which is getting the first title if it exists.
It would be better to write the code in a way that expresses the intention directly, for example:

for item in tree.xpath("//div[@itemprop='itemListElement']"):
    title = get_first_or_empty(item, ".//h3[@itemprop='name']/a/text()")
    phone = get_first_or_empty(item, ".//h4[@itemprop='telephone']/text()")
    print(title, phone)


Where get_first_or_empty is:

def get_first_or_empty(item, xpath):
    matches = item.xpath(xpath)
    if matches:
        return matches[0]
    return ""