Job scheduling for data scraping on Python
Question: I'm scraping (extracting) data from a certain website. The data contains two
values that I need, namely **(grid) frequency value** and **time**.

The data on the website is being updated every second. I'd like to
continuously save these values (append them) into a list or a tuple using
python. To do that I tried using _schedule_ library. The following job
schedule commands run the data scraping function (socket_freq) every second.

    
    
    import schedule
    schedule.every(1).seconds.do(socket_freq)
    
    while True:
        schedule.run_pending()
    

I'm facing two problems:

  1. I don't know how to restrict the schedule to run during a chosen time interval. For example, i'd like to run it for 5 or 10 minutes. how do I define that? I mean how to I tell the schedule to stop after a certain time.
  2. if I run this code and stop it after few seconds (using break), then I often get multiple entries, for example here is one result, where the first list[ ] in the tuple refers to the time value and the second list[ ] is the values of frequency:

out:

    
    
    (['19:27:02','19:27:02','19:27:02','19:27:03','19:27:03','19:27:03','19:27:03','19:27:03','19:27:03','19:27:03','19:27:04','19:27:04','19:27:04', ...], 
    ['50.020','50.020','50.020','50.018','50.018','50.018','50.018','50.018','50.018','50.018','50.017','50.017','50.017'...])
    

As you can see, the time variable is entered (appended) multiple times,
although I used a schedule that runs every 1 second. What i'd actually would
expect to retrieve is:

out:

    
    
    (['19:27:02','19:27:03','19:27:04'],['50.020','50.018','50.017'])
    

Does anybody know how to solve these problems?

Thanks!

(I'm using python 2.7.9)


Answer: Ok, so here's how I would tackle these problems:

  1. Try to obtain a timestamp at the start of your program and then simply check if it has been working long enough each time you execute piece of code you are scheduling.
  2. Use [time.sleep()](https://docs.python.org/2/library/time.html?highlight=time.sleep#time.sleep) to put your program to sleep for a period of time.

Check my example below:

    
    
    import schedule
    import datetime
    import time
    
    # Obtain current time
    start = datetime.datetime.now()
    
    # Simple callable for example
    class DummyClock:
      def __call__(self):
        print datetime.datetime.now()
    
    schedule.every(1).seconds.do(DummyClock())
    
    while True:
        schedule.run_pending()
        # 5 minutes == 300 seconds
        if (datetime.datetime.now() - start).seconds >= 300:
            break
        # And here we halt execution for a second
        time.sleep(1)
    

All refactoring is welcome

