Layering sheens: What kind of paint sheen should I use under a polyurethane semi gloss?
Flat, flat, flat, followed by  whatever you put on the surface. Flat gives you "tooth" for the next layer without you having to sand it (tooth) into a gloss finish.

Though given the described problem, I'd take a careful look at the "latex primer followed by urethane paint" W.R.T.  "weird cracking."