Script to download Annual Reports (~70,000 in total) from a Website It's only 38 lines of code and I haven't worked with web scraping that much before.


  
  How heavy would my code be on their server?
  It's for Deep Learning purposes and I haven't ran it yet but would it result in my IP getting banned a quarter way through once I start
  downloading the 70k pdfs?
  


Also, I'm not sure how efficient this is (right now I don't think I'm even checking if a file exists and I'm overwriting it each time I run my program, so if my code is interrupted halfway through, I'll have to run it again and it'll start downloading everything again from scratch. I'll have to fix that).

But anyways, here is the code:

# Scrapes all pdfs off from www.annualreports.com
# Haven't tested yet but should be somewhere around 70,963 pdfs since my empty search returned 5,479/5,674 stated companies

import requests
from urllib.parse import urljoin
import pandas as pd
from bs4 import BeautifulSoup as bs
import os, re
import pickle

def extract_table():
    r = requests.get('http://www.annualreports.com/Companies?search=')
    soup = bs(r.content, 'lxml')
    df = pd.DataFrame([(i.text, 'http://www.annualreports.com' + i['href']) for i in soup.select('tbody td:nth-of-type(1) a')], columns = ['Company','Link'])
    df.to_pickle('links.pkl') # saves into a dataframe the name of the company plus the href link it points to

def scrap_pdfs():
    df = pd.read_pickle('links.pkl')
    a = 0 # for naming the filenames numerically

    for x in range(df.Link.count()):

        url = df['Link'][x] # reads the "Link" column from the dataframe

        folder_location = r'/home/duke/Annual_Reports/Data' # SPECIFY FULL DIRECTORY

        response = requests.get(url)
        soup= bs(response.text, "html.parser")

        for link in soup.select("a[href$='.pdf']"): #goes through all the .pdfs in all of the href links

            #filename = os.path.join(folder_location, ['href'].split('/')[-1]) # Names the pdf files using the last portion of each link

            filename = os.path.join(folder_location, str(a))
            a+=1

            with open(filename, 'wb') as f:
                f.write(requests.get(urljoin(url,link['href'])).content)
Yes, right now this would be inefficient, I'd definitely say that using
a predictable filename would be the first key change here, if the link
target or description isn't usable as part of the filename, consider
hashing it.  Simply enumerating all links is, well, suboptimal:  What
if, even if previous files are kept around and not overwritten, between
runs the contents of the website changes (seems likely, doesn't it?) and
suddenly the order of the files being download doesn't match what's on
disk?  Definitely do something about that.

Secondly, the code's separating the creation of that pickle file because
it's supposed to be run multiple times?  A bit odd, but as an impromptu
database, why not.  I usually consider pickle files somewhat ephemeral
since the format is (was?) specific to Python.

And then simply check if the file exists and skip it (maybe also check
if it's non-empty; going further before downloading the full data the
HTTP header for the size of the returned content could also be compared,
again, before actually downloading the full files).

The range over the data frame I'd have expected to be simpler, but
right now I can't find if there's an easier way (not using the index,
but simply looping over the values themselves that is).

f.write(requests.get(...).content) - that's gonna buffer the full file
in memory I think.  Better don't do that, instead either requests
might have some facility to write directly to files (or a file-like
object), or alternatively the download content would have to be read
piece-by-piece into a smallish buffer, alternating it with writing that
buffer to the output file.

Finally the formatting could be more consistent (whitespace between
expressions mostly).  Take a look at
PEP8 and perhaps an
autoformatter to do that
automatically.  Also leftover comments should be removed.

Right so overall it's effective, but for production code there's lots of
things that could be done, especially making it a more fully fleshed out
script (read: parse command line arguments, have default parameters for
e.g. the output directory; etc.).  The parsing is fine and the selectors
again are succinct and easy to read.