Compare result files from two different runs the smart way?
Simplify all the things! You and your users will benefit from simplicity.

Option parsing

If the --compare option is either True or False, make it a boolean flag with False as the default value. Your users won't have to remember if the --compare argument is 1, true or yes, they will simply write --compare.

You should also consider moving from optparse to argparse: optparse is deprecated since Python 2.7 and Python 3.2 and could be removed in any future release.

About your assumptions


If the pickle file is moved away from data_output, then the users will simply see that the file is missing. You could catch the resulting exception to ask them to first run the program without comparison.
If the removing fails, then something is really wrong anyway.


I don't think resolving those specific issues is the way to go, and it should be easier for your users to provide this comparison feature differently.

Comparison logic

Your current solution is mostly a hack. It looks like you only asked yourself: what is the simplest way to implement this? The question should be different: what is the simplest way to use this?

Say I want to compare 2010 and 2011. I first run your program with 2010, then with 2011 plus comparison. What if I actually made a typo and typed 2010 instead of 2009? What if one of the options for 2011 was wrong? I have to start all over again. Very frustrating since the processing is slow.

I don't know your program well enough to be sure, but the simplest way could be to provide a --dump option, so that users would dump the first dataset (slow), dump the second dataset (slow), then compare the two (fast!).

It would also be even clearer to provide two binaries, one for processing the datasets, and another one for comparison: this would reduce confusion.

Users are now in control and understand what happens and how the comparison works. Fixing any issue does not require to run processing for the two datasets. User won't delete the dumped files, and if they did, that would be their fault since you exposed those files to them.