Drilling holes to install shelves and came across this material. Is it Asbestos?
It's possible there is some asbestos in that skim coat over the wall. 
 It's impossible to say but it would be totally unnecessary to have this material tested before hanging shelves.  

Asbestos is dangerous when you inhale airborne particles.  It's a real hazard in the workplace, where people may inhale asbestos particles for months or years or decades without the proper precautions.  

The exposure you have when drilling a few holes to install shelves is very small by comparison.  Wear a dust mask when drilling and cleaning up.  By all means, refrain from arranging it into lines on a mirror and snorting it.  

Keep in mind that the silica dust that you make when drilling masonry - any masonry - can also be harmful.  Again the risk with such a limited exposure is small, so it's nothing to get nervous about, but you'll want to take the same precautions against breathing the dust even if there's no asbestos present.