I see cool looking structures underwater sometimes but I never find it worthwhile to try and investigate them - it's dark and I can barely get to the bottom before I need oxygen.

Is it even worth exploring underwater? If it is, what are some effective strategies for doing so?