Should I let pressure treated lumber dry before building with it?
Unless you have a kiln to dry wood in, drying wood in a standard environment takes a really long time (if you buy firewood, usually you want ~2yr old wood!). I don't know that leaving it out for a couple weeks would dry it if it were so saturated that there is visible water coming out of it.  It is recommended to let wood used for hardwoods sit a couple weeks to get accustomed to the environment as far as humidity goes, but it is not being dried any further.

I would recommend that you find a better source of wood that is dried properly.  You also want to carefully select your wood, it might be that you just got a bad piece some how.

For outdoor projects, instead of PT lumber you might opt to try a wood like cedar which naturally resists the elements pretty well.