Eating mangoes with skin intact
It's not about chemicals on the outside of the fruit; washing the fruit well should take care of that. Whether we eat the skin of any given fruit basically boils down to whether it a) tastes good and b) has a pleasant texture. For example, some people eat the skin of the kiwi, despite it having a hairy texture that many people find unpleasant. Many people do not eat the skins of mango because it tastes bitter and has a tough, fibrous texture, but if you don't mind the texture and enjoy that taste, go ahead, enjoy :)  

Note: I'm seeing sources that say if you're particularly sensitive to poison oak, you should not eat mango skins, as they contain one of the chemicals in poison oak but in a much lesser quantity, so it might cause an allergic reaction. You'll probably also break out from touching the mango skin to peel it, so take that as a warning sign. This doesn't apply to the OP, since if you've been eating them all your life you're obviously not allergic, but might apply to other people googling this question.