Is there a formal definition for iteration?
The $k$th iteration is just the $k$th element of some appropriate sequence. In general, I doubt it's a formally defined concept; it's just one of those things whose meaning is basically the natural-language meaning of the word.