Homemade chocolate turns crumbly
GdD gives a good answer, but I am going to give a recommendation type answer based on your ingredients because although the quantities are different the ingredients are exactly the same. Go to Hershey.com and search for their fudge recipe. Follow this recipe.

When your "fudge" reaches nearly the desired consistency from stirring, start stirring in the peanuts, and quickly spoon out your clusters onto wax paper.

I think that will give you the desired results and give you a silky fudge chocolate peanut cluster.