E-commerce scraper needs shortening How would I go about shortening this code?
I'm sure there's a smarter way to write the code.
It's working fine, it's just ugly.

It is a way to scrape the number of items on an e-commerce site. 

import requests
from bs4 import BeautifulSoup
import re

url1 = "https://us.pandora.net/en/charms/?sz=30&amp;start={}&amp;format=page-element"
url2 = "https://us.pandora.net/en/bracelets/?sz=30&amp;start={}&amp;format=page-element"
url3 = "https://us.pandora.net/en/rings/?sz=30&amp;start={}&amp;format=page-element"
url4 = "https://us.pandora.net/en/necklaces/?sz=30&amp;start={}&amp;format=page-element"
url5 = "https://us.pandora.net/en/earrings/?sz=30&amp;start={}&amp;format=page-element"

#res = requests.get(link.format(url1),headers={"User-Agent":"Mozilla/5.0"})

soup1 = BeautifulSoup(requests.get(url1.format(0)).text, 'lxml')
soup2 = BeautifulSoup(requests.get(url2.format(0)).text, 'lxml')
soup3 = BeautifulSoup(requests.get(url3.format(0)).text, 'lxml')
soup4 = BeautifulSoup(requests.get(url4.format(0)).text, 'lxml')
soup5 = BeautifulSoup(requests.get(url5.format(0)).text, 'lxml')

total_items1 = ''.join(re.findall(r'\d', soup1.select_one('span.products-count').text))
total_items2 = ''.join(re.findall(r'\d', soup2.select_one('span.products-count').text))
total_items3 = ''.join(re.findall(r'\d', soup3.select_one('span.products-count').text))
total_items4 = ''.join(re.findall(r'\d', soup4.select_one('span.products-count').text))
total_items5 = ''.join(re.findall(r'\d', soup5.select_one('span.products-count').text))

#categories = [tag['title'].strip() for tag in soup.select('.refinement-link[title]')

#total_items_sale1 = ''.join(re.findall(r'\d', soup1.select_one('.grid-tile .price-standard')))
#total_items_sale1
#total_items_sale1
#total_items_sale1
#total_items_sale1

#print('Categories:')

#for category in categories:
    #print('\t{}'.format(category))
print('\nTotal Charms: {}'.format(total_items1))
print('\nTotal Bracelets: {}'.format(total_items2))
print('\nTotal Rings: {}'.format(total_items3))
print('\nTotal Necklaces: {}'.format(total_items4))
print('\nTotal Earrings: {}'.format(total_items5))
If you find yourself, doing something over and over again, loops are often the answer.

Your code rewritten with a loop:

import re

import requests
from bs4 import BeautifulSoup

urls = {
    "Charms": "https://us.pandora.net/en/charms/?sz=30&amp;start={}&amp;format=page-element",
    "Bracelets": "https://us.pandora.net/en/bracelets/?sz=30&amp;start={}&amp;format=page-element",
    "Rings": "https://us.pandora.net/en/rings/?sz=30&amp;start={}&amp;format=page-element",
    "Necklaces": "https://us.pandora.net/en/necklaces/?sz=30&amp;start={}&amp;format=page-element",
    "Earrings": "https://us.pandora.net/en/earrings/?sz=30&amp;start={}&amp;format=page-element"
}

for category, url in urls.items():
    soup = BeautifulSoup(requests.get(url.format(0)).text, 'lxml')
    total_items = ''.join(re.findall(r'\d', soup.select_one('span.products-count').text))
    print("Total {}: {}".format(category, total_items))


So what has changed? The URLs went into a dictionary where the category's name is the key. A simple list would also work here in your case. Now we can iterate over that dictionary. With for category, url in urls.items(): you get the key, i.e. the name of the category and the URL at the same time.

The loop body now does what you had written out by hand, namely getting the page, parse the content with BeautifulSoup and than again with a regex.

Since the URL came bundled with the category, we can no print all the results nicely without writing the same code over and over again.

You can also go a step further and make this into a function:

import re

import requests
from bs4 import BeautifulSoup


def get_total_items(url):
    """Parse the product count from the given pandora URL"""
    soup = BeautifulSoup(requests.get(url).text, 'lxml')
    return ''.join(re.findall(r'\d', soup.select_one('span.products-count').text))


if __name__ == "__main__":
    urls = {
        "Charms": "https://us.pandora.net/en/charms/?sz=30&amp;start={}&amp;format=page-element",
        "Bracelets": "https://us.pandora.net/en/bracelets/?sz=30&amp;start={}&amp;format=page-element",
        "Rings": "https://us.pandora.net/en/rings/?sz=30&amp;start={}&amp;format=page-element",
        "Necklaces": "https://us.pandora.net/en/necklaces/?sz=30&amp;start={}&amp;format=page-element",
        "Earrings": "https://us.pandora.net/en/earrings/?sz=30&amp;start={}&amp;format=page-element"
    }

    for category, url in urls.items():
        print("Total {}: {}".format(category, get_total_items(url.format(0))))



Now the intermediate variable soup is inside the function and does not clutter the global namespace. You could also now easily change the implementation of get_total_items(...) without the need to adapt the "application" code.

I also added the infamous if __name__ == "__main__":, which is basically Python's way to tell someone looking at the script: "This piece of code is supposed to be run if you use this as a script." Have a look at the official documentation or this Stack Overflow post if you want to learn more.



A more subtle change: I reorderd the imports to follow the recommendation of the official Style Guide for Python Code (often just PEP8), which is to have imports from the standard libraries first, then a blank line, and then all the third party libraries.

In general PEP8 is often worth a read, especially if you are not yet familiar with Python.