Scraping Edgar with Python regular expressions
Question: I am working on a personal project's initial stage of downloading 10-Q
statements from EDGAR. Quick disclaimer, I am very new to programming and
python so the code that I wrote is very basic, not even using custom functions
and classes, just a very long script that I'm more comfortable editing. As a
result, some solutions are quite rough (i.e. concatenating urls using CIKs and
other search options instead of doing requests with "browser" headers)

I keep running into a problem that those who have scraped EDGAR might be
familiar with. Every now and then my script just stops running. It doesn't
raise any exceptions (I created some that append txt reports with links that
can't be opened and so forth). I suspect that either SEC servers have a
certain limit of requests from an IP per some unit of time (if I wait some
time after CTRL-C'ing the script and run it again, it generates more output
compared to rapid re-activation), alternatively it could be TWC that
identifies me as a bot and limits such requests.

If it's SEC, what could potentially work? I tried learning how to work with
TOR and potentially get a new IP every now and then but I can't really find
some basic tutorial that would work for my level of expertise. Maybe someone
can recommend something good on the topic?

Maybe the timers would work? Like force the script to sleep every hour or so
(still trying to figure out how to make such timers and reset them if an event
occurs). The main challenge with this particular problem is that I can't let
it run at night.

Thank you in advance for any advice, I keep fighting with it for days and at
this stage it could take me more than a month to get what I want (before I
even start tackling 10-Ks)


Answer: It seems like delays are pretty useful - sitting at 3.5k downloads with no
interruptions thanks to a simple:

    
    
    import(time)
    
    time.sleep(random.randint(0, 1) + abs(random.normalvariate(0, 0.2)))
    

