Is it safe to overload LEGO train motors?
What could be considered a safe limit that motors can endure for an
  extended period of time?


Their rated voltage.  

Motors are designed to operate at their rated voltage indefinitely, or until they wear out, whichever comes first.  Exceeding this voltage means you shorten the life of the motor.

Your big enemy is heat.  At some point, the amount of heat generated by the windings in the motor will exceed the ability of the motor to dissipate it, and the wire in the winding will melt and break the electrical connection (usually at the point where the wire is soldered to the commutator).  



The point at which failure or excessive stress or wear occurs is influenced by many factors, including the amount of current driving the motor, the speed at which the motor is turning, the amount of load the motor is driving, the ambient air temperature, etc.  These variables make it difficult to predict a safe voltage above the rated one. 

In addition, over-driving motors in this way makes them inefficient; speed increases are offset by higher losses due to back-emf, limiting the additional torque, so driving motors this way eats batteries fast.

Last but not least, overdriven motors are a safety concern.  The paint covering the windings can catch fire and burn down your house; you have to watch such experiments very carefully.

If you really want to increase the power, your best bet is to choose a motor with more torque, so that you can run it at its rated voltage.