Calcium carbonate toxicity
Looking at the Material Safety Data Sheet (MSDS), most of the issues with it seem to stem from the physical consequences of it being in fine dry powder form. This means that if you get it on your skin, it can get into cracks and pores and easily dry out your skin to the point of cracking, and will cause irritation or worse if you get it in your eyes, for example. It's not toxic if you get it on you, though -- rubbing a lump of limestone against your skin isn't going to do anything except abrade it.

So, yes, if you're working with the powder, wear gloves. And safety glasses. And long sleeves. (See also: lab coat.)

If in doubt, follow the MSDS.