Script to download Annual Reports (~70,000 in total) from a Website
Yes, right now this would be inefficient, I'd definitely say that using
a predictable filename would be the first key change here, if the link
target or description isn't usable as part of the filename, consider
hashing it.  Simply enumerating all links is, well, suboptimal:  What
if, even if previous files are kept around and not overwritten, between
runs the contents of the website changes (seems likely, doesn't it?) and
suddenly the order of the files being download doesn't match what's on
disk?  Definitely do something about that.

Secondly, the code's separating the creation of that pickle file because
it's supposed to be run multiple times?  A bit odd, but as an impromptu
database, why not.  I usually consider pickle files somewhat ephemeral
since the format is (was?) specific to Python.

And then simply check if the file exists and skip it (maybe also check
if it's non-empty; going further before downloading the full data the
HTTP header for the size of the returned content could also be compared,
again, before actually downloading the full files).

The range over the data frame I'd have expected to be simpler, but
right now I can't find if there's an easier way (not using the index,
but simply looping over the values themselves that is).

f.write(requests.get(...).content) - that's gonna buffer the full file
in memory I think.  Better don't do that, instead either requests
might have some facility to write directly to files (or a file-like
object), or alternatively the download content would have to be read
piece-by-piece into a smallish buffer, alternating it with writing that
buffer to the output file.

Finally the formatting could be more consistent (whitespace between
expressions mostly).  Take a look at
PEP8 and perhaps an
autoformatter to do that
automatically.  Also leftover comments should be removed.

Right so overall it's effective, but for production code there's lots of
things that could be done, especially making it a more fully fleshed out
script (read: parse command line arguments, have default parameters for
e.g. the output directory; etc.).  The parsing is fine and the selectors
again are succinct and easy to read.