When coding and reviewing code, it's easy to spot places where a design pattern could be used. A chain of command here, a strategy there... It's tempting to dive in and apply patterns even though a better solution might be a switch or some simple if's.

Are there some rules or tips you've found valuable to estimate when to do the actual refactoring?

Wait until it becomes too difficult to add features? Wait until the third time you have to change the code? First time you need a hack?