How strong would a servo motor need to be to power the control surfaces of a fixed wing ultralight aircraft?
First I'd like to say you are thinking outside the box, not a bad thing...

However for an ultralight aircraft this is impractical. The first problem you have is that most ultralight aircraft don't have electrical systems. Adding one may put you over the weight class for an ultralight. Servo's, batteries, joysticks, etc are also not very "ultralight". 

The second thing you need to think about is that R/C's work because a single servo controls two surfaces in the case of ailerons, so they are synchronized. The aerodynamic forces become much, much larger as you scale up, so you will need individual (or multiple) servo's per control surface. Then you need to think about how to synchronize them, and calibrate them.

Next comes in a question of failure modes. If your elevator servo fails, what do you do? What about your electrical system? Dead battery? These FBW systems work on large aircraft because they have multiple redundancies (including traditional cable controls). You won't have the space or weight budget to do that.

It just comes down to trying to shove a complicated system into an uncomplicated aircraft.

And for a little note about "large scale" FBW systems. Usually these systems use hydraulic servo valves, which means that the aircraft includes a hydraulic system (or 3), which are electrically or cable actuated. These can provide sufficient force (along with heavy counterweights) to move the control surfaces. The servo's that you would need would not be available in electrical form since they would need to be powerful, long stroke, and fast. There are servo's that are used in some aircraft for things like flaps, but they are not fast.