Installing wall shelves- Crumbling walls
Walls aren't made out of solid plaster. Plaster is merely a coating. If its hard and 'crumbling' all the way through, my guess is it's a masonry block (or brick) wall coated in plaster. Knowing what region of the world you live in may help us narrow down the potential building technique for you.

If it is masonry, there are two things you need to do:

1) Make sure it's not a party wall. Party walls are fire walls and you may not be allowed to drill into them to begin with.

2) Invest in masonry drill bits and masonry anchor screws.