Start task as soon as multiple tasks are successful I write an hobbyist application that collects soccer-data from different sources around the www and aggregates it for a graphical representation and sends it out as a tweet. Now I want to fully automate this process in a more or less robust way (more or less because 1. this is a hobby and 2. depends on scraped 3rd party websites).

I have problems to wrap my head around how this full automation can be organized. The following tasks need to be done:


check once daily if and when there are soccer matches I am interested in
start collecting data of these matches from all sources as soon as match is 
over
retry collecting data every x minutes if it wasn't available yet (the sources have very different speed in releasing the data and it may take 24hrs. sometimes)
as soon as all sources have been collected successfully launch aggregation and tweet. 


My question:
I don't know how to implement the connection between 3. and 4. in a good way. I think I need some wrapper which is called by the scheduler as soon as a match is over, but this doesn't seem to be a robust way:

class MatchCollector:

    def __init__(self,teams,sources=settings.sources):
        self.teams=teams
        self.sources = sources

    def execute(self):
        finished = False
        counter = 0
        while not finished:
            finished = True
            for source in self.sources:
                s = SourceCollector(source,self.teams)
                successful = s.execute() 
                if not successful:
                    finished = False
            time.sleep(600)
            counter += counter
            if counter == 10000 # arbitrary chosen, have to make calc, when would be a good point to give up
                 raise IterationException('Tried for x hours without success. Somethings broken') 
        d = DataAggregator(sources)
        d.execute()


Am I missing something problematic, by doing this?

I arrange daily scheduling (1., 2.) via cron and python-crontab in a scheduling script. It is started once daily and reads the daily matches and writes again into cron the script to handle the single matches with arguments passed via sys.argv.
Well, whatever you do, you should only re-check those sources which have not finished, so you should remember which have finished already, or equivalently, only those which have not. 

class MatchCollector:
    def __init__(self, teams, sources=settings.sources):
        self.teams = teams
        self.sources = sources
        self.unfinished = {SourceCollector(source, teams) for source in sources}
        self.timeout = 600  # s
        self.iterations = 1000 # arbitrarily chosen, have to make calc, when would be a good point to give up

    def execute(self):
        counter = 0
        while self.unfinished:
            self.unfinished = {source for source in self.unfinished if not s.execute()}
            time.sleep(self.timeout)
            counter += 1
            if counter &gt;= self.iterations 
                 raise IterationException('Tried for x hours without success. Somethings broken') 
        d = DataAggregator(self.sources)
        d.execute()


This assumes that SourceCollector.execute can be run repeatedly without needing to re-initialize it. If this is not the case, just do self.unfinished = {source for source in self.unfinished if not SourceCollector(source, self.teams).execute()}.

Note that doing counter += counter is probably not what you want. Since you initialize it to 0, it does 0 + 0 = 0, so nothing. If you had initialized it to a non-zero value, it would also not simply count up, but double every iteration.

I also made the timeout between retries and the number of iterations members of the class so they are no longer magic numbers.

You should also follow Python's official style-guide, PEP8, which recommends using spaces around = in assignments and after commas in argument lists of functions and methods.