If a player sets a monster that is revealed to be invalid when it is flipped face up, what happens?
In a tournament, the player who made the illegal play could be given a game loss, if the gamestate can't be reconstructed (and sometimes even if it can). Many calls like this are up to the discretion of the judge, and if you're at a larger tournament, you'll be able to appeal to a head judge for a final ruling if you disagree.

Be very careful about this. Even doing something like not activating a mandatory effect (like adding counters to a certain card when something happens) could give you a loss depending on the judge and if the opposing player calls a judge for it. I've heard of people being given game losses for not activating the effects of cards with multiple effects in the order printed on the card (Wavering Eyes in this case, for just the first 2 effects), even though the order wouldn't affect the gamestate in any way.

If you're playing with friends or in a more casual setting, oftentimes people just do the best they can to rewind it, or just play it out.