USB Serial Adapter Unintentionally Powering RS232 TTL Converter
I'm not surprised, inputs usually have some form of protection diode up to the rail which, when the power is off, will conduct. This will be from the Rx and DTR lines and will depend on the state (voltage) of those lines and how much current whatever it is attached to can source.

If it bothers you, then adding some series resistance to the input lines will reduce this effect.