Python Downloading Data File from Web-Scraped URL
Question: I'm trying to develop an automated script to download the following data file
to a utility server and then ETL related processing. Looking for pythonic
suggestions. Not familiar with the current best options for this type of
process between urllib, urllib2, beautiful soup, requests, mechanize,
selenium, etc.

[The Website](http://nppes.viva-it.com/NPI_Files.html)

"Full Replacement Monthly NPI File"

[The Monthly Data File](http://nppes.viva-
it.com/NPPES_Data_Dissemination_March_2015.zip)

The file name (and subsequent url) changes monthly.

Here is my current approach thus far:

    
    
    from bs4 import BeautifulSoup
    import urllib 
    import urllib2
    
    soup = BeautifulSoup(urllib2.urlopen('http://nppes.viva-it.com/NPI_Files.html').read())
    
    download_links = []
    
    for link in soup.findAll(href=True):
        urls = link.get('href', '/')
        download_links.append(urls)
    
    target_url = download_links[2]
    
    urllib.urlretrieve(target_url , "NPI.zip")
    

I am not anticipating the content on this clunky govt. site to change, so I
though just selecting the 3rd element of the scraped url list would be good
enough. Of course, if my entire approach is wrongheaded, I welcome correction
(data analytics is the personal forte). Also, if I am using outdated
libraries, unpythonic practices, or low performance options, I definitely
welcome the newer and better!


Answer: In general [requests](http://docs.python-requests.org/en/latest/) is the
easiest way to get webpages.

If the name of the data files _follows the pattern_
`NPPES_Data_Dissemination_<Month>_<year>.zip`, which seems logical, you can
request that directly;

    
    
    import requests
    
    url = "http://nppes.viva-it.com/NPPES_Data_Dissemination_{}_{}.zip"
    r = requests.get(url.format("March", 2015))
    

The data is then in `r.text`.

If the data-file name is less certain, you can get the webpage and use a
regular expression to search for links to `zip` files;

    
    
    In [1]: import requests
    
    In [2]: r = requests.get('http://nppes.viva-it.com/NPI_Files.html')
    
    In [3]: import re
    
    In [4]: re.findall('http.*NPPES.*\.zip', r.text)
    Out[4]: 
    ['http://nppes.viva-it.com/NPPES_Data_Dissemination_March_2015.zip',
     'http://nppes.viva-it.com/NPPES_Deactivated_NPI_Report_031015.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_030915_031515_Weekly.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_031615_032215_Weekly.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_032315_032915_Weekly.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_033015_040515_Weekly.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_100614_101214_Weekly.zip']
    

The regular expression in In[4] basically says to find strings that start with
"http", contain "NPPES" and end with ".zip". This isn't speficic enough. Let's
change the regular expression as shown below;

    
    
    In [5]: re.findall('http.*NPPES_Data_Dissemination.*\.zip', r.text)
    Out[5]: 
    ['http://nppes.viva-it.com/NPPES_Data_Dissemination_March_2015.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_030915_031515_Weekly.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_031615_032215_Weekly.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_032315_032915_Weekly.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_033015_040515_Weekly.zip',
     'http://nppes.viva-it.com/NPPES_Data_Dissemination_100614_101214_Weekly.zip']
    

This gives us the URLs of the file we want but also the weekly files.

    
    
    In [6]: fileURLS = re.findall('http.*NPPES_Data_Dissemination.*\.zip', r.text)
    

Let's filter out the weekly files:

    
    
    In [7]: [f for f in fileURLS if 'Weekly' not in f]
    Out[7]: ['http://nppes.viva-it.com/NPPES_Data_Dissemination_March_2015.zip']
    

This is the URL you seek. But this whole scheme does depend on how regular the
names are. You can add flags to the regular expression searches to discard the
case of the letters, that would make it accept more.

