What should I be concerned about changing out wheelsets on a Ritchey Cross bike?
So long as the hub width and rim diameter (e.g. a 700c rim is replaced with another 700c rim) are the same, and the rim is designed for rim brakes, there is nothing to be worried about. If you were moving to a larger tire, you'd have to consider clearance, but a smaller tire will clear wherever a larger one will. The new wheel set should have appropriate rim width to support the desired tires. 

The brake alignment won't be a problem assuming you're sticking to the same rim diameter -- the rim width might change a little bit between the wheels, but this is just a tiny re-adjustment of the brakes as usual (if necessary). 

The bike will drop in height a little bit as well if you move to smaller tires -- you wouldn't notice it from 28 -> 25, but if you had 40->20 or something, you'd notice.