How can I check constraints on my state machine behaviour?
Use a model checker.  This is exactly what they're good for.  You might look at SPIN and NuSMV as a starting point, but there are many others as well.