Teaching an old dog new tricks: Finally adopting the micro-service architecture
In your approach I see a few scalability problems - I'm judging from a Google App Engine (GAE) context, where scalability is achieved via breakdown of the work in small tasks/work items and strict limitation of the response time for a task execution:


you're duplicating some piece of functionality: both scripts need to parse the same page, leading to your requirement #1
the task execution time for script A is fundamentally unbound, it can vary widely depending on the number of links discovered which need to be checked if already crawled


I'd approach this a bit differently to avoid these issues. I don't even see a need for separate services for this particular job:


the service pulls individual page URLs as separate tasks/work items
it checks if the URL was already crawled and, if so, does nothing, otherwise continues
gets the page content and parses it to extract 2 kinds of info:


the list of other URLs to crawl, creating and enqueueing new work items for each one of them (it doesn't check here if they were already crawled, which may take a long time depending on the number of URLs)
the other info needed from the page



In this approach the duration of each task execution is reduced to the minimum necessary: one page parsing and only one check if a URL has already been crawled. And each page is only parsed once.

The number of pending work items would be the driver for scaling - if it goes above a level new instances can be started, if it drops below a level and multiple instances are running some may be stopped. If implemented as a GAE app the GAE infra can take care of that for you automatically.

This solution can be implemented using a noSQL database (like Google Datastore, in a GAE context), which can scale better than a SQL one.