Regex doesn't return all img tags - Python
Question: I have a python script that downloads the html and the images shown in the
html so I can open the file locally.

It works fine, the only problem is, there is a certain div in which the images
don't get downloaded/found by the regex. I have no idea why tho. It's not a
huge problem, but I'd like to know the reason.

This is the important part of the script:

    
    
    url = "http://www.somedomain.com"
    urlContent = urllib2.urlopen(url).read()
    #Write originalHtml to file
    f = open("originalHtml",'w')
    f.write(urlContent)
    f.close()
    # HTML image tag: some_text
    imgUrls = re.findall('img .*?src="(.*?)"', urlContent)
    

After that I loop over the links, one by one, downloading the images and
replacing the links in the html so the "src" points to the local path where I
downloaded it. The script takes care of relative links and direct links.

However, part of the images never gets downloaded. This is html that doesn't
get picked up :

    
    
    <img src="/images/news/den-mcx80001.jpg" style="width:60px;height:36px;margin-top:12px; margin-bottom:12px; margin-left:17px; margin-right:17px;float:left; ">
    

This however does get picked up:

    
    
    <img class="productimg" style="width:72px;height:74px;margin-top:15px; margin-bottom:15px; margin-left:3px; margin-right:28px " src="/images/01_prdarticledocs/ImagesSmall/jpr/jpr-prx718xlf.jpg" alt="jpr-prx718xlf">
    

I'm not an expert in regexes, far from it, but it does seem that it should
pick up both, no?


Answer: Fixed with BeautifulSoup, as the comments suggested. Code snippet for anyone
looking for a script to download HTML with the Images, save them and relink
the images in the html to local relative links.

    
    
    import urllib2
    import re
    from BeautifulSoup import BeautifulSoup
    from os.path import basename
    from urlparse import urlsplit
    
    #get content of a url and save (not necessary) the originalhtml
    url = "http://www.someDomain.com"
    urlContent = urllib2.urlopen(url).read()
    page = BeautifulSoup(urlContent)
    f = open("originalHtml",'w')
    f.write(urlContent)
    f.close()
    #Find all images in the file, put them in imgUrls 
    imgUrls = page.findAll('img')
    imagesDict = {}
    
    # download all images
    for image in imgUrls:
        try:
            #get src tag and download file, save link and local link in dict
            imgUrl = image['src']
            imgData = urllib2.urlopen(imgUrl).read()
            fileName = basename(urlsplit(imgUrl)[2])
            location = "images/" + fileName;
            imagesDict[location] = imgUrl
            print "loc=" + location
            output = open(location,'wb')
            output.write(imgData)
            output.close()
        except:
            #not so clean solution to catch hard-linked images ('http://somedomain.com/img/image.jpg
            try:
                imgData = urllib2.urlopen(url + imgUrl).read()
                fileName =  basename(urlsplit(imgUrl)[2])
                location = "images/" + fileName
                imagesDict[location] = imgUrl
                print "loc=" + location
                output = open(location,'wb')
                output.write(imgData)
                output.close()
            except:
                print "Double ERROR"
            print "Error" + imgUrl
            pass
    
    #Replace the old links to new local links
    for dictKey in imagesDict:
        urlContent = re.sub(imagesDict[dictKey], dictKey, urlContent)
    
    
    #save HTML
    f = open("imagesReplaced.html", 'w')
    f.write(urlContent)
    f.close()
    

