Can I dunk a peltier (TEC) module in ethanol?
I'm pretty sure you can't dissolve cured silicone with acetone or ethanol. But this is based on my attention to labels, not actual experience :)

According to this website the only parts you might worry about are the joins between the pellets and the interconnects. However it states "these are either soldered to a metalization on the ceramic or bonded on in a process called Direct Bond Copper (DBC)."

They mention you can use a knife to manually remove the silicone or use a number of different solvents. Personally I'd use a knife to avoid unnecessarily using chemicals.

Edit:

I thought I should also mention that even though the peltier itself should be fine in many solvents the plastic insulation on the leads may not :)