How can I determine the maximum possible current in a transformer?
AFAIK transformers are disigned to withstand a 180% current overload for short periods. (They can be overloaded this way for something like 15 minutes every 24 hours or so.) But in the long run overloading it with 200% rated current will most likely cause serious overheating and eventually damage. Maybe a good (meaning in your case probably water - or oil - cooling, which can be way more expensive than a new transformer) can provide the necessary cooling to enable the trafo to carry that kind of current.

Additional info: If your supply provides 0.1 A DC as you say, the AC current (RMS value) that comes out of the trafo is not the same as the magnitude of the DC current! In general, assuming the output current of the trafo is not distorted (meaning it keeps its beautiful sine shape - not necessarily true) you have for full-wave diode rectifiers (from: Mohan - Power Electronics):

\$I_{out(DC)}=\frac{2}{\pi}\sqrt{2}I_{sin(AC)}=0.9I_{sin(AC)}\$

meaning that the AC current will be actually 0.22 A if you want 0.2 A DC. But, since the trafo is seriously overloaded, I'm sure the current will be strongly distorted due to the saturation of the trafo iron core. At the end of the day, this will lead to extra heat generation and some other odd behavior.

All in all I don't really think it's a very good idea.

The above calculation applies only if you have a line frequency transformer (50 or 60 Hz) followed by a rectifier. If the circuit includes a high frequency transformer, the situation can be entirely different. So, if this is the case, please edit your question, so we can adjust the answer to that.