How hard is it to make fudge?
Like so many things, if you know what you are trying to do, fudge is not difficult to get right.

Fudge is a high fat candy. In the US chocolate is implied but the addition of chocolate doesn't change the process.

You want a solid, creamy candy with a smooth texture.

As Elendil wrote, it is important to ensure that you cook the candy to the correct temperature- but that is only half of the problem.

As the candy cools it will form crystals. In order to have a smooth texture you have to make those crystals as small as possible. The bigger they are the grainier the candy will be. The slower the crystals form the larger they will be able to grow. 

For a smooth texture we want to form crystals as quickly as possible.

Crystals require a trigger to set them off- such as a seed crystal or agitation. If the candy is allowed to cool undisturbed and with no seed sugar crystals in the pot no crystals will form. It will be like a bomb ready to go off at the slightest provocation- this is what we want. After it is cooled we stir it like mad to form all our crystals all at once- fast and small.

Fudge is very forgiving in that it is easy to start over. Unlike an egg custard that when it breaks is unrecoverable- the sugar crystals that ruin fudge are easy to fix. You just add a little water, melt the candy back down, and start over.

Tricks to prevent premature crystallization


Prevent errant crystals!
A lid is placed on the pot for the last few minutes of cooking. This causes condensation to wash any errant sugar crystals off of the sides of the pot. A single errant crystal can ruin your fudge.
Don't agitate the candy
When the candy is cooling it can't be stirred or bumped.
Add distractions
I consider this cheating and it is unnecessary but some recipes reduce the risk of premature crystallization by adding things like corn syrup or marshmallow creme.