Why is my yeast so slow?
There are so many factors in the rising of bread, have a look at this previous answer for something more exhaustive. I will say that 30 minutes is pretty short, especially when you are starting from a cool dough. You are using milk, which will enrich the dough, and although you haven't mentioned it I'm sure there is salt in the mix, both of these will slow yeast development. 

There's nothing wrong with this, in fact slower is better for good gluten and flavor development. I would be prepared to let an enriched potato dough prove at least 3-4 hours, or overnight in the refrigerator. 

24 hours may be too much at it may have over-proofed, also foodborne illnesses may have developed in that time. Personally I'd discard it and try again.