Is designing a small GA airplane with a wing loading of 20-25 lbs/sq ft a bad idea, or is that wing loading too high?
Racing airplanes do that and live with high landing speeds.  25 psf is about the wing loading of a Beech Baron.  The Baron stalls in the landing configuration, gear and flaps down, at 84 mph/73kt.
This mean you fly approaches at 110 mph with flaps.  Not too many GA pilots are safe with an airplane that hot without a lot of extra training and lord help you if you have to put the thing in a field that fast (a single I mean, the Baron being a twin and the expectation of off field landings being low).
You can make the flaps fancier, and maybe get that down under 100 mph, and then add wing slats and maybe get it down to 90, but you just added a lot of weight and cost.  Not worth it for a few knots at the top.