Python is filtering out currency markers
Question: **Objective** : Write a screenscraper that cycles through a selection of web
pages containing old prices and new prices, reads in the prices, and writes
them to a CSV file.

**Method** : The config file urls.txt contains the list of pages. Open that
file and cycle through the URLs. For each URL, use Beautiful Soup to extract
the contents of any divs of class "current-price" and "old-price". Not all
pages will have an old price, so I've made that optional.

**Problem** : It's all working fine but with one curious exception. Where
prices are in dollars, the price and the dollar sign are coming through. Where
prices are in Euros or Pounds Sterling, the currency markers £ and € are being
stripped off. I want the currency markers to come through in all cases. I
suspect it's an encoding issue. (The lstrip calls below are to remove some
errant spaces and tabs that were coming through.)

**Contents of urls.txt:**

    
    
    http://uk.norton.com/norton-security-for-one-device
    http://uk.norton.com/norton-security-antivirus
    http://uk.norton.com/norton-security-with-backup
    http://us.norton.com/norton-security-for-one-device
    http://us.norton.com/norton-security-antivirus
    http://us.norton.com/norton-security-with-backup
    http://ie.norton.com/norton-security-for-one-device
    http://ie.norton.com/norton-security-antivirus
    http://ie.norton.com/norton-security-with-backup
    

**Python code:**

    
    
    ###############################################
    #
    # PRICESCRAPE
    # Screenscraper that checks prices on PD pages
    #
    ###############################################
    
    # Import the modules we need
    import urllib.request
    import re
    import lxml
    from lxml import etree
    from lxml.html.soupparser import fromstring
    from lxml.etree import tostring
    from lxml.cssselect import CSSSelector
    from bs4 import BeautifulSoup, NavigableString
    
    # Open the files we need
    out = open('out.csv', 'w')
    urls=open('urls.txt','r')
    
    # function to take a URL, open the HTML, and return it
    def getPage(url):
        return urllib.request.urlopen(url).read().decode(encoding='UTF-8',errors='strict').encode('ascii','ignore')
    
    out.write('URL,Current Price,Strikethrough Price\n')
    
    
    
    #Loop through the URLs
    for url in urls:
        print('\nExamining ' + url) 
        url=url.rstrip('\n')
        html=getPage(url)
        soup = BeautifulSoup(html,'lxml')
        currentPrice = soup.find('div', {'class': 'current-price'}).contents[0].lstrip('\n').lstrip(' ').lstrip('\t')
        oldPrice = soup.find('div', {'class': 'old-price'}).contents[0].lstrip(' ')
    
        out.write(url)
        out.write(',')
        out.write(str(currentPrice))
        out.write(',')
        if oldPrice:
            out.write(str(oldPrice))
        else:
             out.write('No strikethrough price')
        out.write('\n')
    
        if html =='':
            print('Problem reading page')
    
    print('Done. See out.csv for output')
    
    out.close()
    urls.close()
    


Answer: I would use two modules to make it work and make the code simpler:

  * [`csv`](https://docs.python.org/2/library/csv.html) to export the results into `csv` output file
  * [`requests`](http://docs.python-requests.org/en/latest/) to make the encoding part transparent for you

If you would `import requests` and replace `getPage` implementation with:

    
    
    def getPage(url):
        return requests.get(url).content
    

You would get the prices with the Euro and Pound signs as well.

