Progress report for a long-running process using 'yield' I have a function that processes a large amount of data. I call this function as part of a wider process via a command line script in which many similar but shorter jobs are conducted in sequence.

I have something like this:

import time

def initialise_foo():
    """do something quickly"""
    time.sleep(0.1)

def initialise_bar():
    """do something else quickly"""
    time.sleep(0.1)

def load_data(n_items):
    """do another thing quickly"""
    return range(n_items)    

def process_datum(datum, sleep):
    """"do something with datum that takes a while"""
    time.sleep(sleep)

def cleanup():
    """do a final thing quickly"""
    time.sleep(0.1)


My problem is that I want my script to provide feedback on progress and I was considering converting my data processor into an iterator and using yield to emit progress information.

I can do this but there is a long wait with no feedback...

def process_data1(data, sleep):
    """loop over a lot of data"""
    for item in data:
        process_datum(item, sleep)


def main1(n_items, sleep):
    print("initialising foo")
    initialise_foo()
    print("initialising bar")      
    initialise_bar()
    print("loading data")
    mydata = load_data(n_items)
    print("processing data")
    process_data1(mydata, sleep)
    print("cleaning up")
    cleanup()
    print("done")


So what about this?

def process_data2(data, sleep, report_frequency):
    """loop over a lot of data"""
    for i, item in enumerate(data):
        process_datum(item, sleep)
        if not (i % report_frequency):
            yield i

def main2(n_items, sleep, report_frequency):
    print("initialising foo")
    initialise_foo()
    print("initialising bar")      
    initialise_bar()
    print("loading data")
    mydata = load_data(n_items)
    print("processing data")
    for i in process_data2(mydata, sleep, report_frequency):
        print("%i items processed" % i)
    print("cleaning up")
    cleanup()
    print("done")


run it like this.

if __name__ == "__main__":
    print ("\nwithout progress")
    main1(100, 0.01)
    print ("\nwith progress")
    main2(100, 0.01, 10)


Does this look like a reasonable approach? Any comments on how I have coded this?
I see three basic options you have, to structure progress feedback:


yield
a callback function
inline, hard-coded feedback


Using yield, as you've shown, is one approach.  Ensure you are familiar with Python's implementation of yield and iterables and generators, since several languages share the yield keyword superficially but have subtle implementation differences that might surprise you if, for example, you're used to C#'s yield. Here's a reference on that -- it's subtle but worth reading. Essentially, in Python when your function yields, it returns a generator which can be usefully assigned to a variable, capturing iteration to that point, but you may or may not want to do that.

So there is fundamentally nothing wrong with using yield for this job.  It carries the big advantage of separating pure computing from progress IO.  

A decent alternative to yield which is easy to understand is a callback function.  If you passed a callable as a final parameter to your process_data function then a block of code can be executed, instead of a yield or hardcoded inline feedback, also separating processing from feedback.

One advantage a callback approach carries is it allows communication back to the process_data function which could be used to pause or halt processing.  For example, if your callback function is aware of the console or UI, then while updating progress it can also monitor buttons or keys and return a value to the process_data function.  This could indicate, for example, that the user wants to abort processing and this doesn't pollute the process_data function with awareness of specific UI or IO.  So a callback would also work well and allow you to pass a lambda block or full function in Python.  (Languages without a yield facility may have no choice but to use callbacks.)

Example code:

def process_data(data, sleep, report_frequency, report_callback=None):
    for ...
        ...
        if not (i % report_frequency):  # or whatever criteria
            if report_callback:
                if report_callback(i) == USER_ABORT:  # or other codes, by convention
                  return


Please let me also address what I see as a big, very practical, issue with the code defining report_frequency in terms of data elements or loop iterations.  

I have see two problems with this pattern of progress tracking:


It's difficult to know in advance what a good value of report_frequency is, when it's expressed in loop iterations.  Trial and error can quickly get you into the ballpark, but that leaves problem #2.
Depending on the nature of your computation, the process_datum function may not run in constant time.  For example, it may take longer as you get deeper into the dataset, if numbers are getting larger and depending on what you're calculating.  What I'm getting at is there may not be a constant number of loop iterations per second.


So I'll propose a better way for you to think of report_frequency. Use seconds, not iterations or data elements.  Time is likely what you had in mind, anyways. It's natural to want to see an update every X seconds, regardless of how many loop iterations that is.  This eliminates both problems and is especially useful when your process_datum function does not run in constant time. 

Your code:

def process_data2(data, sleep, report_frequency):
    """loop over a lot of data"""
    for i, item in enumerate(data):
        process_datum(item, sleep)
        if not (i % report_frequency):
            yield i 


Can become this:  (I'm putting the yield vs inline vs callback issue aside for the moment, to illustrate the concept)

def process_data3(data, sleep, report_frequency):
    """loop over a lot of data"""
    t_updlast = datetime.now()
    t_update = t_updlast + timedelta(seconds=report_frequency)    
    for i, item in enumerate(data):
        if datetime.now() &gt;= t_update:
            print("%6.2f%%\n" % (i*100.0/len(data)))
            t_update = datetime.now() + timedelta(seconds=report_frequency)
        process_datum(item, sleep)


Now you'll see a report every report_frequency seconds that looks like:

56.12%
56.83%
...


And this simplifies how you call process_data3 from main. By dropping the yield, you simply call the following from main, without wrapping it in a for loop:

process_data3(mydata, sleep, report_frequency)


And now that you're in the domain of measuring time, if your process_datum function runs in constant time, you can add a fairly simple calculation for "estimated time to completion".  I'll leave that as an exercise for you. :-)

Remember to add this at the top of your module:

from datetime import datetime, timedelta


Sorry for the long answer and timedelta diversion. I hope this is helpful. Good luck!