How to make a scattered dirt effect with an image texture?
The problem you have is that every spot is either a rock texture or a dirt texture. What you need is a way to blend between the two.

The way to do that is to use a mix shader with an image texture plugged into the mix 



The image texture should be a grayscale image - in the areas where it is white you will get 100% rock texture, in the areas where it is black, you will get 100% dirt. Where it is in between, you will get a smooth blend of the two.