Py3 ebay-scraper - Project I need cheap GPU I need cheap GPU's and decided to make a little ebay-scraper to make sure I can buy it at cheap prices. It works and I can search for (any) product to scrape the Title, Price and Shipping cost of that product.

But I feel like I haven't done it in a generic way, hence the need to post it here. Thus my question is, could this have been done more generic?



Code

from bs4 import BeautifulSoup
from urllib.request import urlopen


def find_next_page(soup):
    '''
    Finds next page

    Returns:
        Next page link if exists
        None if next page does not exists
    '''
    next_page = soup.find('a', {'class': 'gspr next'})
    try:
        return next_page['href']
    except:
        return None


def scrape_page(soup, idx):
    '''
    Scrape page for products and save them as a dictionary

    Returns:
        A dictionary of products
    '''
    products = {}
    prod_idx = idx

    for ultag in soup.find_all('ul', {'id': 'ListViewInner'}):

        for litag in ultag.find_all('li'):
            title = litag.find('a', {'class': 'vip'})
            if not title is None:
                products[prod_idx] = {}
                title_text = title.text
                if 'New listing' in title_text:
                    title_text = title_text.replace('New listing', '').lstrip()
                title_text.strip()
                products[prod_idx]['Title'] = title_text

            ul = litag.find('ul', {'class': 'lvprices left space-zero'})
            if not ul is None:
                for li in ul.find_all('li'):
                    if '$' in li.text and not 'shipping' in li.text.lower():
                        products[prod_idx]['Price'] = li.text.split()[0]
                    if 'shipping' in li.text.lower():
                        products[prod_idx]['Shipping'] = li.text.strip()

                prod_idx += 1
    return products, prod_idx


def table_print(products):
    '''
    Prints products in nice human-readable format
    '''
    print ("{:&lt;8} {:&lt;100} {:&lt;15} {:&lt;30}".format('Key', 'Product', 'Price', 'Shipping'))
    for k, v in products.items():
        try:
            t, p, s = v
            print ('{:&lt;8} {:&lt;100} {:&lt;15} {:&lt;30}'.format(k, products[k][t], products[k][p], products[k][s]))
        except ValueError:
            continue


def scrape_product(ebay_page):
    '''
    Main scraper
    '''
    products = {}
    idx = 0

    while not ebay_page is None:
        html_doc = urlopen(ebay_page)
        soup = BeautifulSoup(html_doc, 'html.parser')
        prod, idx = scrape_page(soup, idx)
        products.update(prod)
        ebay_page = find_next_page(soup)

    return products


def make_search_string(keywords):
    '''
    Make ebay search products string
    '''
    base = 'https://www.ebay.com/sch/i.html?_from=R40&amp;_trksid=m570.l1313&amp;_nkw='
    end = '&amp;_sacat=0'
    return '{0}{1}{2}'.format(base, '+'.join(keywords), end)

if __name__ == '__main__':
    print ('Project I Need a Cheap GPU (Ebay-Scraper in Python3) \n@Ludisposed \n')
    keywords = input('What do you want to search ebay for? ').split()

    # Make search string
    ebay_page = make_search_string(keywords)

    # Find all products
    products = scrape_product(ebay_page)

    # Print all products
    table_print(products)




Example

Project I Need a Cheap GPU (Ebay-Scraper in Python3) 
@Ludisposed 

What do you want to search ebay for? amd 580
Key      Product                                                                                              Price           Shipping                      
0        XFX AMD Radeon RX 580 8GB GDDR5 GTR Black Edition PCI Express 3.0 New Sealed                         $305.00         +$37.83 shipping              
1        MSI AMD Radeon RX 580 GAMING X 4G GDDR5 DVI/2HDMI/2Displayport PCI-Express Video                     $336.99         +$46.19 shipping   
etc....
There are multiple things to improve, let's try to categorize them.

Code Style


there is a number of PEP8 code style violations, specifically:


documentations strings need to be enclosed into triple double quotes, start with a capital letter and end with a dot (reference)
group imports properly

handling a bare exception with except: is considered a bad practice
when you check for a substring to be present in a string, you can use not in, e.g. not 'shipping' in li.text.lower() can be replaced with a more readable:

'shipping' not in li.text.lower()

I would also improve the way you format the search URL, extracting the base url as a constant and using str.format() to insert keywords into the URL:

BASE_URL_TEMPLATE = 'https://www.ebay.com/sch/i.html?_from=R40&amp;_trksid=m570.l1313&amp;_nkw={keywords}&amp;_sacat=0'


def make_search_string(keywords):
    """Constructs a search products string."""
    return BASE_URL_TEMPLATE.format(keywords='+'.join(keywords))



Web-Scraping and HTML Parsing


if you would switch to requests instead of urllib, you then can seriously improve on the performance of making the requests by re-using a session:


  ..if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase.

lxml if used instead of html.parser can provide an HTML parsing performance boost as well
see if using a SoupStrainer to focus your HTML parser on the desired part of the HTML tree only is also applicable in your scenario. It can further speed up the HTML parsing step  
using a LBYL approach inside the find_next_page would be a bit more readable and concise:

def find_next_page(soup):
    """
    Finds next page.

    Returns:
        Next page link if exists
        None if next page does not exists
    """
    next_page_link = soup.select_one('a.next')
    return next_page_link['href'] if next_page_link else None

scrape_page() function is overly complicated. For instance, there is a single element with id="ListViewInner" on the page and it does not require a loop to locate this element. And, you can do better in locating elements on the page:

def scrape_page(soup, index):
    """
    Scrapes page for products and save them as a dictionary.

    Returns:
        A dictionary of products
    """
    products = {}

    for produce_index, item in enumerate(soup.select('#ResultSetItems li'), index):
        title = item.select_one('a.vip')
        price = item.select_one(".prc")
        shipping_price = item.select_one('.ship')

        products[produce_index] = {
            'Title': title.find(text=True, recursive=False) if title else None,
            'Price': price.get_text(strip=True) if price else None,
            'Shipping': shipping_price.get_text(strip=True).split()[0] if shipping_price else None
        }

    return products, produce_index


(not tested)


Also note the use of enumerate() for better handling enumeration (thanks @graipher for the idea).