Why do glider licences have fewer restrictions than powered aircraft?
First of all, gliders fly slowly. Everything happens more slowly than in other aircraft, and this helps to make flying them less challenging than you might think. Flying them well is a different matter - this requires a lot of skill and good physical health.
Historically, the early age limits were introduced in Nazi Germany when the Deutscher Luftstportverband (DLSV) and later the NSFK organized groups of adolescents to let them build and fly gliders. Flight training started at the age of 14, mostly in very simple, bungee-launched gliders like the Zögling or SG-38 which were flown in summer and built or repaired in Winter. Since bungee launching is a very labor-intensive process and flights last only a few seconds, this was less about pilot training than keeping the young occupied and under the influence of the Nazi party.

Bungee launch of a Zögling glider (picture source)
And it worked. Accidents did happen, but were almost never fatal since all operations happened at low speed and altitude. Therefore, these age limits were exported together with the idea of flying unpowered aircraft.
Note that in Europe the same health certificate is required for glider flying as for flying GA aircraft.
Now to your concerns about the demanding nature of gliding:

Glider flying is done from airfields which are suitable for GA aircraft (or the gliders could not be aerotowed) or winch launching. 800 meters is about the minimum, and this leaves plenty of space to land a glider.
Flight training is done initially only in the direct vicinity of the airfield, and students are supposed to stay within a cone which allows them to reach the airfield at any moment. Flying cross-country comes much later, when the students have made more than a hundred landings and know how to get the plane down in a certain spot. Again, the slow speed helps to make such landings simple if the pilot is prepared.
Today, training is initially in two-seaters, so there is an experienced pilot around for the first landings and the first cross-country flights.
Aerotowing is also quite simple, and altitude management while in tow is more challenging than keeping the right distance from the towplane. Again, this is done initially in two-seaters, and all the glider pilot has to do is to fly with the same bank angle as and slightly above the towplane. The rope takes care of the rest.