Parsing xml in python - don't understand the DOM
Question: I've been reading up on parsing xml with python all day, but looking at the
site i need to extract data on, i'm not sure if i'm barking up the wrong tree.
Basically i want to get the 13-digit barcodes from a supermarket website
(found in the name of the images). For example:

<http://www.tesco.com/groceries/SpecialOffers/SpecialOfferDetail/Default.aspx?promoId=A31033985>

has 11 items and 11 images, the barcode for the first item is 0000003235676.
However when i look at the page source (i assume this is the best way to
extract all of the barcodes in one go with python, urllib and beautifulsoup)
all of the barcodes are on one line (line 12) however the data doesn't seem to
be structured as i would expect in terms of elements and attributes.

    
    
        new TESCO.sites.UI.entities.Product({name:"Lb Mens Mattifying Dust 7G",xsiType:"QuantityOnlyProduct",productId:"275303365",baseProductId:"72617958",quantity:1,isPermanentlyUnavailable:true,imageURL:"http://img.tesco.com/Groceries/pi/805/5021320051805/IDShot_90x90.jpg",maxQuantity:99,maxGroupQuantity:0,bulkBuyLimitGroupId:"",increment:1,price:2.5,abbr:"g",unitPrice:3.58,catchWeight:"0",shelfName:"Mens Styling",superdepartment:"Health & Beauty",superdepartmentID:"TO_1448953606"});
    new TESCO.sites.UI.entities.Product({name:"Lb Mens Thickening Shampoo 250Ml",xsiType:"QuantityOnlyProduct",productId:"275301223",baseProductId:"72617751",quantity:1,isPermanentlyUnavailable:true,imageURL:"http://img.tesco.com/Groceries/pi/225/5021320051225/IDShot_90x90.jpg",maxQuantity:99,maxGroupQuantity:0,bulkBuyLimitGroupId:"",increment:1,price:2.5,abbr:"ml",unitPrice:1,catchWeight:"0",shelfName:"Mens Shampoo ",superdepartment:"Health & Beauty",superdepartmentID:"TO_1448953606"});
    new TESCO.sites.UI.entities.Product({name:"Lb Mens Sculpting Puty 75Ml",xsiType:"QuantityOnlyProduct",productId:"275301557",baseProductId:"72617906",quantity:1,isPermanentlyUnavailable:true,imageURL:"http://img.tesco.com/Groceries/pi/287/5021320051287/IDShot_90x90.jpg",maxQuantity:99,maxGroupQuantity:0,bulkBuyLimitGroupId:"",increment:1,price:2.5,abbr:"ml",unitPrice:3.34,catchWeight:"0",shelfName:"Pastes, Putty, Gums, Pomades",superdepartment:"Health & Beauty",superdepartmentID:"TO_1448953606"});
    

Maybe something like BeautifulSoup is overkill? I understand the DOM tree is
not the same thing as the raw source, but why are they so different - when i
go to inspect element in firefox the data seems structured as i would expect.

Apologies if this comes across as totally stupid, thanks in advance.


Answer: Unfortunately, the barcode is not given in the HTML as structured data; it
only appears embedded as part of a URL. So we'll need to isolate the URL and
then pick off the barcode with string manipulation:

    
    
    import urllib2
    import bs4 as bs
    import re
    import urlparse
    
    url = 'http://www.tesco.com/groceries/SpecialOffers/SpecialOfferDetail/Default.aspx?promoId=A31033985'
    
    response = urllib2.urlopen(url)
    content = response.read()
    # with open('/tmp/test.html', 'w') as f:
    #     f.write(content)
    # Useful for debugging off-line:
    # with open('/tmp/test.html', 'r') as f:
    #     content = f.read()
    soup = bs.BeautifulSoup(content)
    barcodes = set()
    for tag in soup.find_all('img', {'src': re.compile(r'/pi/')}):
        href = tag['src']
        scheme, netloc, path, query, fragment = urlparse.urlsplit(href)
        barcodes.add(path.split('\\')[1])
    
    print(barcodes)
    

yields

    
    
    set(['0000003222737', '0000010039670', '0000010036297', '0000010008393', '0000003050453', '0000010062951', '0000003239438', '0000010078402', '0000010016312', '0000003235676', '0000003203132'])
    

