Why am I finding condensation in my dryer?
Usually a dryer vent will have a spring loaded flap where it exits the house to prevent outside air (and bugs, rodents, etc) from coming in through the pipe.  If you are getting airflow from outside through your vent pipe, it sounds like either you don't have one or it isn't closing.  I would check that first.