Do I need to be concerned about moisture around dryer exhaust?
I think your venting is leaking.  If it was not leaking, you would not be having condensation.  Condensation in this area is a result of warm moist air - from the dryer - contacting a cool surface.  Seal the leak, solve the problem.

I've been working on dryer professionally for 40 years.