Webscraping Bing wallpapers I wanted to scrape all the wallpapers from the Bing wallpaper gallery.  This was for personal use and to learn about webscraping.  The gallery progressively gets images using javascript as the user scrolls down the page.  Then you can download the high-resolution image with a few clicks.  I actually succeeded in scraping all the wallpapers, but I sort of "cheated" and wanted to know if there's a better way.

Here's what I did:

First, I went to the site in the browser, started capturing network traffic with developer tools, and scrolled all the way down the page as far as possible.  Then I exported the log to a .csv file.  This produced a bunch of URLs like:


  http://az619519.vo.msecnd.net/files/PolandFarm_EN-US8019728095_320x180.jpg


Luckily, the full resolution version (for almost every image) is located in the same place, except with "1920x1200" substituted at the end:


  http://az619519.vo.msecnd.net/files/PolandFarm_EN-US8019728095_1920x1200.jpg


I ran the following script to retrieve the full resolution images:

import csv
import os
import re
import urllib.request

target_dir = 'C:/Users/[Username]/Desktop/Bing'

def main():
    os.chdir(target_dir)
    with open('NetworkData.csv', 'r', encoding='utf-8') as csvfile:
        for line in csv.reader(csvfile):
            url_request = repr(line[0])
            if r'vo.msecnd.net/files' in url_request:
                pic_url = re.sub('_\d+x\d+\.jpg', '_1920x1200.jpg', url_request)
                pic_url = pic_url.strip('\'"')
                try:
                    urllib.request.urlretrieve(pic_url, pic_url.split('/')[-1])
                except:
                    print("Image not found.")

if __name__ == '__main__':
    main()


I thought this solution was actually okay, but I was wondering if there was a way to automate or avoid the manual step of capturing the network traffic.  I was hoping to avoid selenium if possible, because it looks kind of slow and cumbersome.  Also, if you have any advice on improving the script, that would be good.  I had some issues with string encoding, and I feel like in the end the script isn't very readable.
Since you only use one function from both re and os, you could just import the specific function from each rather than the whole module.

from os import chdir
from re import sub


You shouldn't use a bare try except. You might be ignoring all sorts of errors without noticing. Instead, use the exception that you're expecting is likely to happen. According to this Stack Overflow answer urlretrieve will likely raise a URLError (which you also need to get with import URLError from urllib2). This way if you're having other problems you'll actually know about it.

try:
    urllib.request.urlretrieve(pic_url, pic_url.split('/')[-1])
except URLError:
    print("Image not found.")


Also it seems silly to have that message print out without indicating which images weren't found. You may be incorrectly parsing a lot of images without realising it. Just adding (pic_url.split('/')[-1] + " Image not found") would make it a much more useful error message. But it would be even more helpful if you logged the full path (pic_url).