Normal map on only part of the material
There were a couple of things I changed with your file, so here's my edited version: Edited .blend

First off there were a few things you did wrong with your normal map:


When using an image for the normal map you have to set the Texture node to non-color data mode.
Then, you don't plug the image texture into the Displacement input of the material node(only takes B/W data), but instead into the Normal input of your respective shader.


Now that we've covered that it becomes very easy to achieve what you want:

You plug the normal map only into your pebble shader, so that only the pebbles get bump mapping effects, and the mask you already created does the rest. For that I added a second vector input to you group nodes so that they can receive normal mapping data as well.

Also, if you are using only one UV map, a single UV map node is usually enough.